NOTICE

Copyright 2024 Arc Institute. All rights reserved
Copyright 2024 Michael Poli. All rights reserved
Copyright 2024 Stanford University. All rights reserved

This project incorporates and modifies the components below:


- Added training and inference support for Hyena2
- Scaled model training (up to 40B) and context windows (up to 1M)
- Biological/genomic inference and generation evals/tasks 

See AUTHORS file for list of contributing authors.

See LICENSE file for software license. This software is licensed under the Apache License, Version 2.0.
======================================================================

StripedHyena
Copyright 2023-2024 Together
Project URL: https://github.com/togethercomputer/stripedhyena

This project includes software developed at Together for deep signal processing, hybrid architecture composed of rotary (grouped) attention and gated convolutions arranged in Hyena blocks, with improved scaling over decoder-only Transformers.

======================================================================

GPT-NeoX
Copyright 2021-2024 EleutherAI and contributors 
Project URL: https://github.com/EleutherAI/gpt-neox 

This project includes software developed at EleutherAI for large-scale language model training and inference. 

======================================================================

Megatron-LM
Copyright 2019-2024 NVIDIA Corporation
Project URL: https://github.com/NVIDIA/Megatron-LM

This project includes software developed at NVIDIA Corporation for large-scale transformer model training.

======================================================================

DeepSpeed
Copyright 2020-2024 Microsoft Corporation
Project URL: https://github.com/microsoft/DeepSpeed

This project includes software developed at Microsoft Corporation as part of the DeepSpeed deep learning optimization library.

======================================================================