Downloading the model

Some models from the huggingface hub require a login to download. You can use the command huggingface-cli login to login to your account. If you do not have an account, you can create one here. Once you have an account, you can use the command huggingface-cli login to login to your account and download the model, or you can use a token to download the model. You can find more information about the token in this link. Follow the instructions to create an access token, and make sure to check the box: Read access to contents of all public gated repos you can access. After saving, copy the token and store it in a safe location. For reference, we will use the name HF_TOKEN going forward in this section. In a terminal on a login node in one of the DRAC clusters, export your token with the command below (we will use this terminal going forward with the installation of the model):

export HF_TOKEN=<the-hugging-face-token> 

To use the Llama 3.2 model, you need to request permission to download the model. First, you need to visit the Huggingface hub page to provide your personal information and agree to the terms of use. Go to this link to provide your information. Once you have provided your information, you will receive an email with the confirmation of your request if accepted. Once you have received the confirmation, you can use the huggingface-cli to download the model.

To download a model, you can follow the instructions provided in the Llama-3.2 model page on hugginface hub in this link. To follow the steps in this section more easily, you can use the scripts below to create the virtual environment, install the dependencies, and download the model. The script will install the huggingface-cli and download the model using the token provided in the HF_TOKEN variable. You should run the script in a terminal on a login node in one of the DRAC clusters.

The script llama-env.sh (also found in this link) will create a virtual environment and install the required packages. Investigate the script and find the keyword arguments. Notice that the bash script accepts two arguments to set the virtual environment location and the virtual environment name. This script will be used by a number of other scripts in this section. Make sure all scripts are stored at the same location, and with the correct names.

#!/bin/bash
set -e

# Default environment name
ENVNAME=".llama-env" 

# Default location for the environment
ENVLOC="$SLURM_TMPDIR" 
[ -n "$ENVLOC" ] || ENVLOC="$HOME"

# Parse arguments
while [[ "$#" -gt 0 ]]; do
    case $1 in
        --envname|-e) ENVNAME="$2"; shift ;;
        --envloc|-l) ENVLOC="$2"; shift ;;
        *) echo "Unknown parameter passed: $1"; exit 1 ;;
    esac
    shift
done

# Load the required modules
module load cuda cudnn python

# Create the virtual environment
virtualenv --no-download --seeder=pip $ENVLOC/$ENVNAME

# Activate the virtual environment
source $ENVLOC/$ENVNAME/bin/activate

# Upgrade pip
pip install --no-index --upgrade pip

# Install the required packages
pip install --no-index transformers torch

# Export the environment variables
export ENVNAME
export ENVLOC

The script drac-llama-download.sh (also found in this link) will download the model using the huggingface-cli. The script will create a folder with the name of the model, and download the model in that folder. The script will also create a folder to store the output of the model. The script will use the HF_TOKEN variable to download the model. Make sure to run this script in the same location as the llama-env.sh script. Notice the keyword arguments accepted in this script (you can use the --help option to see the available options: bash drac-llama-download.sh --help). Notice the new options to store the models to be downloaded.

#!/bin/bash
# drac-llama-download.sh
set -e

# Set the hugging face token
HF_TOKEN=""

# Directory to save the model
MODELDIR="$HOME/models" 

# Model name
LLAMA_MODEL="Llama-3.2-3B" 

# Parse keyword arguments for hugging face token
while [[ "$#" -gt 0 ]]; do
    case $1 in
        -t|--hf-token) 
            if [[ -z "$2" ]]; then
            echo "Error: --hf-token requires a value."
            exit 1
            fi
            HF_TOKEN="$2"; shift ;;
        -m|--modeldir) MODELDIR="$2"; shift ;;
        -n|--modelname) LLAMA_MODEL="$2"; shift ;;
        -h|--help)
            echo "Usage: $0 --hf-token <hugging_face_token>"
            echo ""
            echo "Options:"
            echo "  -t, --hf-token    Hugging Face token for authentication (required)"
            echo "  -m, --modeldir    Path to the directory where models are stored (default: $HOME/models)"
            echo "  -n, --modelname   Name of the Llama model to download (default: Llama-3.2-3B)"
            echo "  -h, --help        Display this help message"
            exit 0
            ;;
        *) echo "Unknown parameter passed: $1"; exit 1 ;;
    esac
    shift
done

# Ensure the token are provided
if [[ -z "$HF_TOKEN" ]]; then
    echo "Error: --hf-token is required."
    exit 1
fi

# The name of the environment to create
ENVNAME="llama-env-download" 

# Create the environment and install basic dependencies
source llama-env.sh --envname $ENVNAME --envloc $HOME

echo "HF_TOKEN: $HF_TOKEN"
echo "MODELDIR: $MODELDIR"
echo "LLAMA_MODEL: $LLAMA_MODEL"
echo "ENVNAME: $ENVNAME"

# Install the huggingface-hub CLI
pip install --upgrade huggingface-hub[cli]

# Download the model using huggingface-cli
# huggingface-cli download meta-llama/$LLAMA_MODEL --include "original/*" --local-dir $MODELDIR/$LLAMA_MODEL --token $HF_TOKEN 
huggingface-cli download meta-llama/$LLAMA_MODEL --local-dir $MODELDIR/$LLAMA_MODEL --token $HF_TOKEN 

# Clean up
deactivate
rm -rf $ENVLOC/$ENVNAME
echo "Finished successfully"

Now that you have both llama-env.sh and drac-llama-download.sh scripts, and access to your personal HF_TOKEN, you can run the script download the model. Some of the models require a considerable amount of memory. You can check the available memory for your account using the diskusage_report command. Make sure that the model location is large enough to accomodate the models. If you are using the $HOME directory for all content in this chapter, you will probably not have enough space, and therefore, you should choose another location to store the models. A good choice for the location could be your project or scratch storage (scratch storage is usually larger, but files will be purged after a few weeks). You can find more information about the storage in this link.

Select a location to store the models, then copy this location to use in your algorithms. You can export the location to a variable, or use it directly in the command line. For example, if you want to store the models in your $HOME/models directory, you can use the following command:

export MODEL_LOCATION=$HOME/models

If you intend on using your scratch storage (the one we will use moving forward), you can use the variable $SCRATCH. For example, if you want to store the models in your $SCRATCH/models directory, you can use the following command:

export MODEL_LOCATION=$SCRATCH/models

Now that you have the model location choosen and set, we can download the models with the command below (notice that we have to provide the model name: <model-name>):

bash drac-llama-download.sh --modeldir $MODEL_LOCATION --modelname <model-name> --hf-token $HF_TOKEN

Go ahead and download the model Llama-3.2-3B (this process can take a while):

bash drac-llama-download.sh --modeldir $MODEL_LOCATION --modelname Llama-3.2-3B --hf-token $HF_TOKEN

Go ahead and download the model Llama-3.2-3B-Instruct (this process can take a while):

bash drac-llama-download.sh --modeldir $MODEL_LOCATION --modelname Llama-3.2-3B-Instruct --hf-token $HF_TOKEN

Now, take a look at the contents of the folder, and notice the two new folders with the weights and other information about the models.

The inference script

To run the python script for the llama model inference, you can use the script drac-llama.py (also found in this link link). This script will load the model and generate text based on the input provided. Inspect the code for the available arguments.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import os
import argparse
from transformers import pipeline
import json


homepath = os.getenv("HOME")

parser = argparse.ArgumentParser(description="Generate text using a LLaMA model.")
parser.add_argument("-m","--models_path",type=str,default=f"{homepath}/models",help="Path to the model directory.")
parser.add_argument("-n","--model_name",type=str,default="Llama-3.2-3B",help="Name of the model to use.")
parser.add_argument("-p","--prompt",type=str,default="Tell a story in more than 300 but less than 500 words. The story should start with: Once upon a time, in a digital research alliance cluster, a group of researchers embarked on a quest to harness the power of artificial intelligence. They were",help="Prompt to start the text generation.")
parser.add_argument("-s","--source_prompts_file",type=str,default=None,help="Path to the source prompt file. File should be a list with a list of dictionaries with keys 'role' and 'content'.")
parser.add_argument("-o","--output_file",type=str,default=None,help="Path to the output file to save the generated text.")
parser.add_argument("-l","--max_new_tokens",type=int,default=1024,help="Maximum number of new tokens to generate.")

def main():
    print("Drac LLaMA Text Generation")
    print("===================================")
    args = parser.parse_args()
    print(args) 

    pipe = pipeline(
        "text-generation", 
        model=os.path.join(args.models_path, args.model_name),
        torch_dtype=torch.bfloat16, 
    )
    
    output_content = []
    
    if args.source_prompts_file is not None:
        # Load the source prompt file
        with open(args.source_prompts_file, 'r') as f:
            source_prompts = json.load(f)
            
        output_content = pipe(source_prompts, max_new_tokens=args.max_new_tokens)
        print(f"Input Prompt: {source_prompts}")
        print("Generated Text:")
        print(output_content)

    else:
        output_content = pipe(args.prompt, max_new_tokens=args.max_new_tokens, num_return_sequences=1, return_full_text=False)
        print(f"Input Prompt: {args.prompt}")
        print("Generated Text:")
        print(output_content)
    
    if args.output_file is not None:
        try:
            with open(args.output_file, 'w', encoding='utf-8') as f:
                json.dump(output_content, f, ensure_ascii=False, indent=4)
                print(f"Output saved to {args.output_file}")
        except Exception as e:
            print(f"Error writing to output file: {e}")


if __name__ == "__main__":
    main()

python drac-llama.py --models_path $SCRATCH/models --model_name Llama-3.2-3B --prompt "What is the capital of Canada?" --output_file "capital_of_canada.json"

LLama-3.2-3B-Instruct

The Llama-3.2-3B-Instruct model is a variant of the Llama model that is specifically designed for instruction-based tasks. This model is trained to follow instructions and generate text based on the provided input. The main difference between the Llama-3.2-3B and the Llama-3.2-3B-Instruct models is that the latter is optimized for understanding and executing instructions, making it more suitable for tasks that require following specific guidelines or prompts.

The Llama-3.2-3B-Instruct model can be used for various applications, including:

• Task-oriented Dialogues: Engaging in conversations where the model needs to follow specific instructions or guidelines.
• Question Answering: Providing answers to questions based on the given context or instructions.
• Text Generation with Constraints: Generating text that adheres to specific constraints or requirements.
• Instruction Following: Executing tasks based on user-provided instructions.
• Interactive Applications: Enhancing user interactions by following instructions and providing relevant responses.

The Instruct model uses specific tokens to better understand the context of user requirements. The model can be used with a pipeline for text generation that accepts these tokens. In addition, you can use a chat template to generate the text. The example below shows an example of chat template that could be used as an input to the drac-llama.py script:

[
  {
    "role": "system",
    "content": "You are a story teller that builds up on a starting sequence. You should limit the story size to be between 300 and 500 words."
  },
  {
    "role": "user",
    "content": " Once upon a time, in a digital research alliance cluster, a group of researchers embarked on a quest to harness the power of artificial intelligence. They were"
  }
]

With the chat template above, you can run the drac-llama.py script with the following command:

python drac-llama.py --models_path $SCRATCH/models --model_name Llama-3.2-3B-Instruct --output_file "output_chat_template.json" --source_prompts_file "sample_input.json"

The output will be saved in the file output_chat_template.json. The output will contain the generated text based on the input provided in the chat template. The output will be similar to the one below:

Sbatch script

The script below (drac-llama.sh, also found in this link) will be used to submit the job to the DRAC resources. The script will load the required modules, set up the environment, and run the drac-llama.py script using the python command. The script will load the model weights for the Llama model from the storage location provided. The script will then run the drac-llama.py script and generate text based on the input provided.

#!/bin/bash
#SBATCH --time=1:0:0
#SBATCH --mem-per-gpu=12G
#SBATCH --gpus-per-node=1
#SBATCH --nodes=1

# The model directory
MODELDIR="$HOME/models"
# The name of the model to be used (previously downloaded)
LLAMA_MODEL="Llama-3.2-3B"
# The script to be executed
SCRIPT="drac-llama.py"
# Additional parameters to be passed to the SCRIPT
SCRIPTKWARGS=""

# Parse optional keyword arguments
while [[ "$#" -gt 0 ]]; do
    case $1 in
        --modeldir) MODELDIR="$2"; shift ;;
        --llama-model) LLAMA_MODEL="$2"; shift ;;
        --script) SCRIPT="$2"; shift ;;
        --scriptkwargs) SCRIPTKWARGS="$2"; shift ;;
        --help)
            echo "Usage: $0 [--modeldir <path>] [--llama-model <model_name>]"
            echo "  --modeldir       Path to the directory where models are stored (default: $HOME/models)"
            echo "  --llama-model    Name of the Llama model to download (default: Llama-3.2-3B)"
            echo "  --script         Name of the Python script to execute (default: drac-llama.py)"
            echo "  --scriptkwargs   Additional arguments to pass to the script"
            exit 0 ;;
        *) echo "Unknown parameter passed: $1"; exit 1 ;;
    esac
    shift
done

# Check if the variables are set, otherwise set them to default values
[ -n "$ENVNAME" ] || ENVNAME=".llama-env"
[ -n "$ENVLOC" ] || ENVLOC="$SLURM_TMPDIR"
[ -n "$ENVLOC" ] || ENVLOC="$HOME"

# Create the environment and install basic dependencies
source llama-env.sh --envname $ENVNAME --envloc $ENVLOC

# Print information for debugging
printf -- '-%0.s' {1..100} | xargs echo 
echo "Virtual environment: $ENVLOC/$ENVNAME"
echo "Model directory: $MODELDIR"
echo "Llama model: $LLAMA_MODEL"
echo "Script: $SCRIPT"
printf -- '-%0.s' {1..100} | xargs echo 

# Run the python script with the specified keyword arguments
python $SCRIPT $SCRIPTKWARGS \
    --models_path $MODELDIR \
    --model_name $LLAMA_MODEL 

# Cleanup
deactivate
rm -rf $ENVLOC/$ENVNAME

The script allows you to provide a number of keyword arguments. Take a look at the script and identify the keyword arguments provided. To run the script with the following configurations:

• Model name: Llama-3.2-3B-Instruct
• Model location: $SCRATCH/models
• Input chat: sample_input.json
• Output file: output_chat.json
• Max new tokens: 500

You can use the command below to submit the job to the DRAC resources:

sbatch -A def-<someuser> drac-llama.sh \
--modeldir $SCRATCH/models \
--llama-model Llama-3.2-3B-Instruct \
--scriptkwargs "--source_prompts_file sample_input.json --output_file output_chat.json --max_new_tokens 500"

Notice that the account --def-<someuser> was provided, but if you have only one account, you can omit this setting. The output (output_chat.json) will be similar to the one below (notice the last entry, where the role is assistant):