How to run Llam-2 or Llama-3 ?

[kent-yao]

According to https://github.com/amd/RyzenAI-SW/blob/main/example/transformers/models/llm/docs/README.md ,

To run "python run_awq.py --model_name llama-2-7b-chat --task quantize"

Error message below:
######################################################################################
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.2.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last): File "Z:\models\llm\run_awq.py", line 12, in
import llm_eval
File "Z:\tools\llm_eval.py", line 16, in
import torch
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch_init_.py", line 1382, in
from .functional import * # noqa: F403
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch\functional.py", line 7, in
import torch.nn.functional as F
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch\nn_init_.py", line 1, in
from .modules import * # noqa: F403
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch\nn\modules_init_.py", line 35, in
from .transformer import TransformerEncoder, TransformerDecoder,
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch\nn\modules\transformer.py", line 20, in
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\torch\nn\modules\transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
Token has not been saved to git credential helper. Pass add_to_git_credential=True if you want to set the git credential as well.
Token is valid (permission: fineGrained).
Your token has been saved to C:\Users\kent.cache\huggingface\token
Login successful
Namespace(model_name='llama-2-7b-chat', target='aie', profile_layer=False, task='quantize', precision='w4abf16', flash_attention_plus=False, profilegemm=False, dataset='raw', fast_mlp=False, fast_attention=False, w_bit=4, group_size=128, algorithm='awq', gen_onnx_nodes=False, mhaops='all')
Traceback (most recent call last):
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\utils_errors.py", line 304, in hf_raise_for_status
response.raise_for_status()
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\requests\models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/llama-2-7b-chat/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\transformers\utils\hub.py", line 385, in cached_file
resolved_file = hf_hub_download(
^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\file_download.py", line 1403, in hf_hub_download
raise head_call_error
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\file_download.py", line 1261, in hf_hub_download
metadata = get_hf_file_metadata(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\utils_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\file_download.py", line 1667, in get_hf_file_metadata
r = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\file_download.py", line 385, in _request_wrapper
response = _request_wrapper(
^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\file_download.py", line 409, in _request_wrapper
hf_raise_for_status(response)
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\huggingface_hub\utils_errors.py", line 352, in hf_raise_for_status
raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-676bbd2a-7d092399373bffa84d8f94a4;c408a963-ee2a-466d-8ccd-51d3bfa6e2e7)

Repository Not Found for url: https://huggingface.co/llama-2-7b-chat/resolve/main/config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "Z:\models\llm\run_awq.py", line 346, in
model = CausalLMModel.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\transformers\modeling_utils.py", line 2926, in from_pretrained
resolved_config_file = cached_file(
^^^^^^^^^^^^
File "C:\Users\kent\anaconda3\envs\ryzenai-transformers\Lib\site-packages\transformers\utils\hub.py", line 406, in cached_file
raise EnvironmentError(
OSError: llama-2-7b-chat is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>
###################################################################################################

[poganesh]

From the above log, I see two issues. To resolve the issues:

1. Downgrade NumPy:
   pip install numpy==1.26.4

2. The script is looking for llama-2-7b-chat in the local folder and cannot find it. Download the original model using the correct model name locally before running the command to quantize it.

   cd <transformers/models/llm>

   a. Ensure you're authenticated:
   huggingface-cli login

   b. Download the model:
   - For Llama-2-7b-chat:
   huggingface-cli download meta-llama/Llama-2-7b-chat-hf --local-dir ./llama2-7b-chat
   - For Llama-2-7b:
   huggingface-cli download meta-llama/Llama-2-7b-hf --local-dir ./llama2-7b

   c. Quantize the model:
   - For Llama-2-7b-chat:
   python run_awq.py --model_name llama-2-7b-chat --task quantize
   - For Llama-2-7b:
   python run_awq.py --model_name llama-2-7b --task quantize