publish finished auto exl2 script

finally after hours of writing, troubleshooting, and testing across multiple days of low motivation, I finally finish this (hopefully no bugs). I may add more comments though and a modified version for local quants, maybe some updates for more features as well.

README.md

@@ -16,17 +16,18 @@ Feel free to send in PRs or use this code however you'd like.\
 
 - [Manage branches (create/delete)](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/manage%20branches.py)
 
+- [Auto EXL2 upload](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/auto-exl2-upload/auto-exl2-upload.zip?download=true)
+
 - [EXL2 Single Quant V3](https://colab.research.google.com/drive/1Vc7d6JU3Z35OVHmtuMuhT830THJnzNfS?usp=sharing) **(COLAB)**
 
-- [EXL2 Local Quant Windows](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/exl2-windows-local/exl2-windows-local.zip?download=true)
+- [EXL2 Local Quant - Windows](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/exl2-windows-local/exl2-windows-local.zip?download=true)
 
-- [Upload folder to repo](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/upload%20folder%20to%20repo.py)
+- [Upload folder to HF](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/upload%20folder%20to%20repo.py)
 
 ## work in progress/not tested ([unfinished](https://huggingface.co/Anthonyg5005/hf-scripts/tree/unfinished) branch)
 
-- Auto exl2 upload script
-  - Will create repo and create a specified number of custom quants on individual branches
-  - Windows/Linux support
+- EXL2 Multi Quant local
+  - Will replace Local Quant Windows on this readme, and have Linux support. Modified version of Auto EXL2 upload without the upload and deleting part.
 
 ## other recommended stuff
 
@@ -38,19 +39,22 @@ Feel free to send in PRs or use this code however you'd like.\
 ## usage
 
 - Manage branches
-  - Run script and follow prompts. You will be required to be logged in to HF Hub. If you are not logged in, you will need a WRITE token. You can get one in your [HuggingFace settings](https://huggingface.co/settings/tokens). May get some updates in the future for handling more situations. All active updates will be on the [unfinished](https://huggingface.co/Anthonyg5005/hf-scripts/tree/unfinished) branch. Colab and Kaggle keys are supported.
+  - Run script and follow prompts. You will be required to be logged in to HF Hub. If you are not logged in, you will need a WRITE token. You can get one in your [HuggingFace settings](https://huggingface.co/settings/tokens). Colab and Kaggle secret keys are supported.
 
-- EXL2 Single Quant
-  - Allows you to quantize to exl2 using colab. This version creates a exl2 quant to upload to private repo. Should work on any Linux jupyterlab server with CUDA, ROCM should be supported by exl2 but not tested. Only 7B tested on colab.
+- Auto EXL2 upload
+  - This script is designed to automate the process of quantizing models to EXL2 and uploading them to the HF Hub as seperate branches. This is both available to run on Windows and Linux.
 
 - EXL2 Local Quant Windows
-  - Easily creates environment to quantize models to exl2 using Windows to your local machine.
+  - Easily creates environment to quantize models to exl2 using Windows to your local machine. Replacing soon.
 
 - Upload folder to repo
-  - Uploads user specified folder to specified repo, can create private repos too. Not the same as git commit and push, instead uploads any additional files.
+  - Uploads user specified folder to specified repo, can create private repos too. Not the same as git commit and push, instead uploads any additional files. This is more to be modified to your needs then used by itself.
+
+- EXL2 Single Quant
+  - Allows you to quantize to exl2 using colab. This version creates a exl2 quant to upload to private repo. Only 7B tested on colab.
   
 - Download models
-  - Make sure you have [huggingface_hub](https://pypi.org/project/huggingface-hub/) installed as it has the same dependencies. You can install it with '`pip install huggingface-hub`'. To use the script, open a terminal and run '`python download-model.py USER/MODEL:BRANCH`'. There's also a '`--help`' flag to show the available arguments. To download from private repositories, make sure to login using '`huggingface-cli login`' or (not recommended) `HF_TOKEN` environment variable.
+  - To use the script, open a terminal and run '`python download-model.py USER/MODEL:BRANCH`'. There's also a '`--help`' flag to show the available arguments. To download from private repositories, make sure to login using '`huggingface-cli login`' or (not recommended) `HF_TOKEN` environment variable.
 
 ## extras
 

auto-exl2-upload/INSTRUCTIONS.txt

@@ -0,0 +1,38 @@
+For NVIDIA cards install the CUDA toolkit
+
+Nvidia Maxwell or higher
+https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64
+
+Nvidia Kepler or higher
+https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows&target_arch=x86_64
+
+
+Haven't done much testing but for Windows, Visual Studio with desktop development for C++ might be required. I've gotten cl.exe errors on a previous install
+
+
+This may work with AMD cards but only on linux. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
+
+
+
+First setup your environment by using either windows.bat or linux.sh. If something fails during setup, then every file and folder except for windows.bat, linux.sh, and exl2-quant.py should be deleted then try again.
+
+After setup is complete then you'll have a file called start-quant. Use this to run the quant script.
+
+Make sure that your storage space is 3x the amount of the model's size. To mesure this, take the number of billion parameters and mutliply by two, afterwards mutliply by 3 and that's the recommended storage. There's a chance you may get away with 2.5x the size as well.
+Make sure to also have a lot of RAM depending on the model.
+
+If you close the terminal or the terminal crashes, check the last BPW it was on and enter the remaining quants you wanted. It should be able to pick up where it left off. Don't type the finished BPW as it will start from the beginning. You may also use ctrl + c pause at any time during the quant process.
+
+Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+
+
+Credit to turboderp for creating exllamav2 and the exl2 quantization method.
+https://github.com/turboderp
+
+Credit to oobabooga the original download and safetensors scripts.
+https://github.com/oobabooga
+
+Credit to Lucain Pouget for maintaining huggingface-hub.
+https://github.com/Wauplin
+
+Only tested with CUDA 12.1 on Windows 11 and half-tested Linux through WSL2 but I don't have enough RAM to fully test but quantization did start.

auto-exl2-upload/auto-exl2-upload.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb52c9577ce88784c0831237f91a9d3edcdcafb3c85ec94b8dac85e036daf8a4
+size 6692

auto-exl2-upload/exl2-quant.py

@@ -0,0 +1,190 @@
+#usually it's on the inside that counts, not this time. This script is a mess, but it works.
+#import required modules
+from huggingface_hub import login, get_token, whoami, repo_exists, file_exists, upload_folder, create_repo, upload_file, create_branch
+import os
+import sys
+import subprocess
+import glob
+
+#define os differences
+oname = os.name
+if oname == 'nt':
+    osclear = 'cls'
+    osmv = 'move'
+    osrmd = 'rmdir /s /q'
+    oscp = 'copy'
+    pyt = 'venv\\scripts\\python.exe'
+    slsh = '\\'
+elif oname == 'posix':
+    osclear = 'clear'
+    osmv = 'mv'
+    osrmd = 'rm -r'
+    oscp = 'cp'
+    pyt = './venv/bin/python'
+    slsh = '/'
+else:
+    sys.exit('This script is not compatible with your machine.')
+def clear_screen():
+    os.system(osclear)
+
+#get token
+if os.environ.get('KAGGLE_KERNEL_RUN_TYPE', None) is not None: #check if user in kaggle
+    from kaggle_secrets import UserSecretsClient # type: ignore
+    from kaggle_web_client import BackendError # type: ignore
+    try:
+        login(UserSecretsClient().get_secret("HF_TOKEN")) #login if token secret found
+    except BackendError: 
+        print('''
+            When using Kaggle, make sure to use the secret key HF_TOKEN with a 'WRITE' token.
+                   This will prevent the need to login every time you run the script.
+                   Set your secrets with the secrets add-on on the top of the screen.
+             ''')
+if get_token() is not None:
+    #if the token is found then log in:
+    login(get_token())
+    tfound = "Where are my doritos?" #doesn't matter what this is, only false is used
+else:
+    #if the token is not found then prompt user to provide it:
+    login(input("API token not detected. Enter your HuggingFace (WRITE) token: "))
+    tfound = "false"
+
+#if the token is read only then prompt user to provide a write token:
+while True:
+    if whoami().get('auth', {}).get('accessToken', {}).get('role', None) != 'write':
+        clear_screen()
+        if os.environ.get('HF_TOKEN', None) is not None: #if environ finds HF_TOKEN as read-only then display following text and exit:
+            print('''
+                  You have the environment variable HF_TOKEN set.
+                                 You cannot log in.
+          Either set the environment variable to a 'WRITE' token or remove it.
+                  ''')
+            input("Press enter to continue.")
+            sys.exit("Exiting...")
+        if os.environ.get('COLAB_BACKEND_VERSION', None) is not None:
+            print('''
+                              Your Colab secret key is read-only
+                Please switch your key to 'write' or disable notebook access on the left.
+                  ''')
+            sys.exit("Stuck in loop, exiting...")
+        elif os.environ.get('KAGGLE_KERNEL_RUN_TYPE', None) is not None:
+            print('''
+                                      Your Kaggle secret key is read-only
+                Please switch your key to 'write' or unattach from notebook in add-ons at the top.
+                          Having a read-only key attched will require login every time.
+                ''')
+        print("You do not have write access to this repository. Please use a valid token with (WRITE) access.")
+        login(input("Enter your HuggingFace (WRITE) token: "))
+        continue
+    break
+clear_screen()
+
+#get original model repo url
+repo_url = input("Enter unquantized model repository (User/Repo): ")
+
+#look for repo
+if repo_exists(repo_url) == False:
+    print(f"Model repo doesn't exist at https://huggingface.co/{repo_url}")
+    sys.exit("Exiting...")
+model = repo_url.replace("/", "_")
+modelname = repo_url.split("/")[1]
+clear_screen()
+
+#ask for number of quants
+qmount = int(input("Enter the number of quants you want to create: "))
+qmount += 1
+clear_screen()
+
+#save bpw values
+print(f"Type the BPW for the following {qmount - 1} quants. Recommend staying over 2.4 BPW. Use the vram calculator to find the best BPW values: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator")
+qnum = {}
+for i in range(1, qmount):
+    qnum[f"bpw{i}"] = float(input(f"Enter BPW for quant {i} (2.00-8.00): ")) #convert input to float for proper sorting
+clear_screen()
+
+#collect all values in a list for sorting
+bpwvalue = list(qnum.values())
+
+#sort the list from smallest to largest
+bpwvalue.sort()
+
+if not os.path.exists(f"models{slsh}{model}{slsh}converted-st"): #check if model was converted to safetensors, skip download if it was
+    result = subprocess.run(f"{pyt} download-model.py {repo_url}", shell=True) #download model from hf (Credit to oobabooga for this script)
+    if result.returncode != 0:
+        print("Download failed.")
+        sys.exit("Exiting...")
+    clear_screen()
+
+if not glob.glob(f"models/{model}/*.safetensors"): #check if safetensors model exists
+    convertst = input("Couldn't find safetensors model, do you want to convert to safetensors? (y/n): ")
+    while convertst != 'y' and convertst != 'n':
+        convertst = input("Please enter 'y' or 'n': ")
+    if convertst == 'y':
+        print("Converting weights to safetensors, please wait...")
+        result = subprocess.run(f"{pyt} convert-to-safetensors.py models{slsh}{model} --output models{slsh}{model}-st", shell=True) #convert to safetensors (Credit to oobabooga for this script as well)
+        if result.returncode != 0:
+            print("Converting failed. Please look for a safetensors model or convert model manually.")
+            sys.exit("Exiting...")
+        subprocess.run(f"{osrmd} models{slsh}{model}", shell=True)
+        subprocess.run(f"{osmv} models{slsh}{model}-st models{slsh}{model}", shell=True)
+        open(f"models{slsh}{model}{slsh}converted-st", 'w').close()
+        print("Finished converting")
+    else:
+        sys.exit("Can't quantize a non-safetensors model. Exiting...")
+clear_screen()
+
+#create new repo if one doesn't already exist
+if repo_exists(f"{whoami().get('name', None)}/{modelname}-exl2") == False:
+    print("Creating model repository...")
+    create_repo(f"{whoami().get('name', None)}/{modelname}-exl2", private=True)
+    print(f"Created repo at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2") #notify user of repo creation
+
+    #create the markdown file
+    print("Writing model card...")
+    with open('./README.md', 'w') as file:
+        file.write(f"# Exl2 quants for [{modelname}](https://huggingface.co/{repo_url})\n\n")
+        file.write("## Automatically quantized using the auto quant from [hf-scripts](https://huggingface.co/anthonyg5005/hf-scripts)\n\n")
+        file.write(f"Would recommend {whoami().get('name', None)} to change up this README to include more info.\n\n")
+        file.write("### BPW:\n\n")
+        for bpw in bpwvalue:
+            file.write(f"[{bpw}](https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2/tree/{bpw}bpw)\n\n")
+    print("Created README.md")
+
+    upload_file(path_or_fileobj="README.md", path_in_repo="README.md", repo_id=f"{whoami().get('name', None)}/{modelname}-exl2", commit_message="Add temp README") #upload md file
+    print("Uploaded README.md to main")
+else:
+    input("repo already exists, are you resuming a previous process? (Press enter to continue, ctrl+c to exit)")
+
+#start converting
+for bpw in bpwvalue:
+    if os.path.exists(f"{model}-measure{slsh}measurement.json"): # Check if measurement.json exists
+        cmdir = False
+        mskip = f" -m {model}-measure{slsh}measurement.json" #skip measurement if it exists
+    else:
+        cmdir = True
+        mskip = ""
+    print(f"Starting quantization for BPW {bpw}")
+    os.makedirs(f"{model}-exl2-{bpw}bpw-WD", exist_ok=True) #create working directory
+    os.makedirs(f"{model}-exl2-{bpw}bpw", exist_ok=True) #create compile full directory
+    subprocess.run(f"{oscp} models{slsh}{model}{slsh}config.json {model}-exl2-{bpw}bpw-WD", shell=True) #copy config to working directory
+    #more settings exist in the convert.py script, to veiw them go to docs/convert.md or https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
+    result = subprocess.run(f"{pyt} exllamav2/convert.py -i models/{model} -o {model}-exl2-{bpw}bpw-WD -cf {model}-exl2-{bpw}bpw -b {bpw}{mskip}", shell=True) #run quantization and exit if failed (Credit to turbo for his dedication to exl2)
+    if result.returncode != 0:
+        print("Quantization failed.")
+        sys.exit("Exiting...")
+    if cmdir == True:
+        os.makedirs(f"{model}-measure", exist_ok=True) #create measurement directory
+        subprocess.run(f"{oscp} {model}-exl2-{bpw}bpw-WD{slsh}measurement.json {model}-measure", shell=True) #copy measurement to measure directory
+        open(f"{model}-measure/Delete folder when no more quants are needed from this model", 'w').close()
+    try:
+        create_branch(f"{whoami().get('name', None)}/{modelname}-exl2", branch=f"{bpw}bpw") #create branch
+    except:
+        print(f"Branch {bpw} already exists, trying upload...")
+    upload_folder(folder_path=f"{model}-exl2-{bpw}bpw", repo_id=f"{whoami().get('name', None)}/{modelname}-exl2", commit_message=f"Add quant for BPW {bpw}", revision=f"{bpw}bpw") #upload quantized model
+    subprocess.run(f"{osrmd} {model}-exl2-{bpw}bpw-WD", shell=True) #remove working directory
+    subprocess.run(f"{osrmd} {model}-exl2-{bpw}bpw", shell=True) #remove compile directory
+
+if file_exists(f"{whoami().get('name', None)}/{modelname}-exl2", "measurement.json") == False: #check if measurement.json exists in main
+    upload_file(path_or_fileobj=f"{model}-measure{slsh}measurement.json", path_in_repo="measurement.json", repo_id=f"{whoami().get('name', None)}/{modelname}-exl2", commit_message="Add measurement.json") #upload measurement.json to main
+    
+print(f'''Quants available at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2
+      \nRepo is private, go to https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2/settings to make public if you'd like.''')

auto-exl2-upload/linux-setup.sh

@@ -0,0 +1,63 @@
+#!/bin/bash
+
+# converted from windows-setup.bat by github copilot
+
+# check if "venv" subdirectory exists, if not, create one
+if [ ! -d "venv" ]; then
+    python -m venv venv
+else
+    echo "venv directory already exists. If something is broken, delete everything but exl2-quant.py and run this script again."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# ask if the user has git installed
+read -p "Do you have git and wget installed? (y/n) " gitwget
+
+if [ "$gitwget" = "y" ]; then
+    echo "Setting up environment"
+else
+    echo "Please install git and wget before running this script."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8. If ROCm, install pytorch for ROCm 5.7
+read -p "Please enter your GPU compute version, CUDA 11/12 or AMD ROCm (11, 12, rocm): " pytorch_version
+
+if [ "$pytorch_version" = "11" ]; then
+    echo "Installing PyTorch for CUDA 11.8"
+    venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cu118
+elif [ "$pytorch_version" = "12" ]; then
+    echo "Installing PyTorch for CUDA 12.1"
+    venv/bin/python -m pip install torch
+elif [ "$pytorch_version" = "rocm" ]; then
+    echo "Installing PyTorch for AMD ROCm 5.7"
+    venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/rocm5.7
+else
+    echo "Invalid compute version. Please enter 11, 12, or rocm."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# download stuff
+echo "Downloading files"
+git clone https://github.com/turboderp/exllamav2
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/convert-to-safetensors.py
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/download-model.py
+
+echo "Installing pip packages"
+
+venv/bin/python -m pip install -r exllamav2/requirements.txt
+venv/bin/python -m pip install huggingface-hub transformers accelerate
+venv/bin/python -m pip install ./exllamav2
+
+# create start-quant.sh
+echo "#!/bin/bash" > start-quant.sh
+echo "venv/bin/python exl2-quant.py" >> start-quant.sh
+echo "read -p \"Press enter to continue\""
+chmod +x start-quant.sh
+echo "If you use ctrl+c to stop, you may need to also use 'pkill python' to stop running scripts."
+echo "Environment setup complete. run start-quant.sh to start the quantization process."
+read -p "Press enter to exit"
+exit

auto-exl2-upload/windows-setup.bat

@@ -0,0 +1,63 @@
+@echo off
+
+setlocal
+
+REM check if "venv" subdirectory exists, if not, create one
+if not exist "venv\" (
+    python -m venv venv
+) else (
+    echo venv directory already exists. If something is broken, delete everything but exl2-quant.py and run this script again.
+    pause
+    exit
+)
+
+REM ask if the user has git installed
+set /p gitwget="Do you have git and wget installed? (y/n) "
+
+if "%gitwget%"=="y" (
+    echo "Setting up environment"
+) else (
+    echo Please install git and wget before running this script.
+    echo winget install wget
+    echo winget install git
+    pause
+    exit
+)
+
+REM if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8
+echo CUDA path: %CUDA_HOME%
+set /p cuda_version="Please enter your CUDA version (11 or 12): "
+
+if "%cuda_version%"=="11" (
+    echo Installing PyTorch for CUDA 11.8...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu118
+) else if "%cuda_version%"=="12" (
+    echo Installing PyTorch for CUDA 12.1...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu121
+) else (
+    echo Invalid CUDA version. Please enter 11 or 12.
+    pause
+    exit
+)
+
+REM download stuff
+echo Downloading files...
+git clone https://github.com/turboderp/exllamav2
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/convert-to-safetensors.py
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/download-model.py
+
+echo Installing pip packages...
+
+venv\scripts\python.exe -m pip install -r exllamav2/requirements.txt
+venv\scripts\python.exe -m pip install huggingface-hub transformers accelerate
+venv\scripts\python.exe -m pip install .\exllamav2
+
+REM create start-quant-windows.bat
+echo @echo off > start-quant.bat
+echo venv\scripts\python.exe exl2-quant.py >> start-quant.bat
+echo REM tada sound for fun >> start-quant.bat
+echo powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync(); >> start-quant.bat
+echo pause >> start-quant.bat
+powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync();
+echo Environment setup complete. run start-quant.bat to start the quantization process.
+pause

exl2-windows-local/instructions.txt

@@ -12,4 +12,10 @@ make sure you setup the environment by using windows-setup.bat
 after everything is done just download a model using download-model.bat
 to quant, use convert-model-auto.bat. Enter the model's folder name, then the BPW for the model
 
-You can always pause the quantization process by pressing Ctrl + C and typing exit. All progress will be stored in the WD (working directory) folder. You can resume where you left off by running the convert-model-auto.bat script with the same arguments you used before.
+You can always pause the quantization process by pressing Ctrl + C and typing exit. All progress will be stored in the WD (working directory) folder. You can resume where you left off by running the convert-model-auto.bat script with the same arguments you used before.
+
+Credit to turboderp for creating exllamav2 and the exl2 quantization method.
+https://github.com/turboderp
+
+Credit to oobabooga the original download script.
+https://github.com/oobabooga

ipynb/Multi_Quant_exl2.ipynb

@@ -0,0 +1,255 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "7PhL3HkpFeU7"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Setup environment\n",
+        "#@markdown Takes about 15 minutes to finish\n",
+        "# download stuff\n",
+        "!git clone https://github.com/turboderp/exllamav2\n",
+        "!wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/convert-to-safetensors.py\n",
+        "!wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/download-model.py\n",
+        "!pip install -r exllamav2/requirements.txt\n",
+        "!pip install huggingface-hub transformers accelerate --upgrade\n",
+        "!pip install ./exllamav2"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "CXbUzOmNHyff"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Login to Huggingface - Required\n",
+        "#import required functions\n",
+        "import os\n",
+        "import sys\n",
+        "from huggingface_hub import login, get_token, whoami\n",
+        "\n",
+        "#get token\n",
+        "if os.environ.get('KAGGLE_KERNEL_RUN_TYPE', None) is not None: #check if user in kaggle\n",
+        "    from kaggle_secrets import UserSecretsClient # type: ignore\n",
+        "    from kaggle_web_client import BackendError # type: ignore\n",
+        "    try:\n",
+        "        login(UserSecretsClient().get_secret(\"HF_TOKEN\")) #login if token secret found\n",
+        "    except BackendError:\n",
+        "        print('''\n",
+        "            When using Kaggle, make sure to use the secret key HF_TOKEN with a 'WRITE' token.\n",
+        "                   This will prevent the need to login every time you run the script.\n",
+        "                   Set your secrets with the secrets add-on on the top of the screen.\n",
+        "             ''')\n",
+        "if get_token() is not None:\n",
+        "    #if the token is found then log in:\n",
+        "    login(get_token())\n",
+        "else:\n",
+        "    #if the token is not found then prompt user to provide it:\n",
+        "    login(input(\"API token not detected. Enter your HuggingFace (WRITE) token: \"))\n",
+        "\n",
+        "#if the token is read only then prompt user to provide a write token (Only required if user needs a WRITE token, remove if READ is enough):\n",
+        "while True:\n",
+        "    if whoami().get('auth', {}).get('accessToken', {}).get('role', None) != 'write':\n",
+        "        if os.environ.get('HF_TOKEN', None) is not None: #if environ finds HF_TOKEN as read-only then display following text and exit:\n",
+        "            print('''\n",
+        "          You have the environment variable HF_TOKEN set.\n",
+        "          You cannot log in.\n",
+        "          Either set the environment variable to a 'WRITE' token or remove it.\n",
+        "                  ''')\n",
+        "            input(\"Press enter to continue.\")\n",
+        "            sys.exit(\"Exiting...\")\n",
+        "        if os.environ.get('COLAB_BACKEND_VERSION', None) is not None:\n",
+        "            print('''\n",
+        "                              Your Colab secret key is read-only\n",
+        "                Please switch your key to 'write' or disable notebook access on the left.\n",
+        "                  ''')\n",
+        "            sys.exit(\"Stuck in a loop, exiting...\")\n",
+        "        elif os.environ.get('KAGGLE_KERNEL_RUN_TYPE', None) is not None:\n",
+        "            print('''\n",
+        "                                      Your Kaggle secret key is read-only\n",
+        "                Please switch your key to 'write' or unattach from notebook in add-ons at the top.\n",
+        "                          Having a read-only key attched will require login every time.\n",
+        "                ''')\n",
+        "        print(\"You do not have write access to this repository. Please use a valid token with (WRITE) access.\")\n",
+        "        login(input(\"Enter your HuggingFace (WRITE) token: \"))\n",
+        "        continue\n",
+        "    break"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "dxKEA7obHLoO"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Start quant\n",
+        "#@markdown ### Using subprocess to execute scripts doesn't output on Colab. If something seems frozen, please wait. Any detected errors will automatically stop Colab\n",
+        "#import required modules\n",
+        "from huggingface_hub import login, get_token, whoami, repo_exists, model_info, upload_folder, create_repo, upload_file, create_branch\n",
+        "import os\n",
+        "import sys\n",
+        "import subprocess\n",
+        "import glob\n",
+        "\n",
+        "#define os differences\n",
+        "oname = os.name\n",
+        "if oname == 'nt':\n",
+        "    osmv = 'move'\n",
+        "    osrmd = 'rmdir /s /q'\n",
+        "    oscp = 'copy'\n",
+        "    pyt = 'venv\\\\scripts\\\\python.exe'\n",
+        "    slsh = '\\\\'\n",
+        "elif oname == 'posix':\n",
+        "    osmv = 'mv'\n",
+        "    osrmd = 'rm -r'\n",
+        "    oscp = 'cp'\n",
+        "    pyt = 'python'\n",
+        "    slsh = '/'\n",
+        "else:\n",
+        "    sys.exit('This script is not compatible with your machine.')\n",
+        "\n",
+        "#get original model repo url\n",
+        "#@markdown Enter unquantized model repository (User/Repo):\n",
+        "repo_url = \"mistralai/Mistral-7B-Instruct-v0.2\" # @param {type:\"string\"}\n",
+        "\n",
+        "#look for repo\n",
+        "if repo_exists(repo_url) == False:\n",
+        "    print(f\"Model repo doesn't exist at https://huggingface.co/{repo_url}\")\n",
+        "    sys.exit(\"Exiting...\")\n",
+        "model = repo_url.replace(\"/\", \"_\")\n",
+        "modelname = repo_url.split(\"/\")[1]\n",
+        "print(\"\\n\\n\")\n",
+        "\n",
+        "#ask for number of quants\n",
+        "#@markdown Enter the number of quants you want to create:\n",
+        "quant_amount = \"5\" # @param {type:\"string\"}\n",
+        "qmount = int(quant_amount)\n",
+        "qmount += 1\n",
+        "\n",
+        "#save bpw values\n",
+        "#@markdown You will be asked the BPW values after running this section.\n",
+        "print(f\"Type the BPW for the following {qmount - 1} quants. Recommend staying over 2.4 BPW. Use the vram calculator to find the best BPW values: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator\")\n",
+        "qnum = {}\n",
+        "for i in range(1, qmount):\n",
+        "    qnum[f\"bpw{i}\"] = float(input(f\"Enter BPW for quant {i} (2.00-8.00): \")) #convert input to float for proper sorting\n",
+        "print(\"\\n\\n\")\n",
+        "\n",
+        "#collect all values in a list for sorting\n",
+        "bpwvalue = list(qnum.values())\n",
+        "\n",
+        "#sort the list from smallest to largest\n",
+        "bpwvalue.sort()\n",
+        "\n",
+        "if not os.path.exists(f\"models{slsh}{model}{slsh}converted-st\"): #check if model was converted to safetensors, skip download if it was\n",
+        "    print(\"Starting download...\")\n",
+        "    result = subprocess.run(f\"{pyt} download-model.py {repo_url}\", shell=True) #download model from hf (Credit to oobabooga for this script)\n",
+        "    if result.returncode != 0:\n",
+        "        print(\"Download failed.\")\n",
+        "        sys.exit(\"Exiting...\")\n",
+        "    print(\"Download finished\\n\\n\")\n",
+        "\n",
+        "#@markdown You will also be asked to convert model to safetensors if needed\n",
+        "if not glob.glob(f\"models/{model}/*.safetensors\"): #check if safetensors model exists\n",
+        "    convertst = input(\"Couldn't find safetensors model, do you want to convert to safetensors? (y/n): \")\n",
+        "    while convertst != 'y' and convertst != 'n':\n",
+        "        convertst = input(\"Please enter 'y' or 'n': \")\n",
+        "    if convertst == 'y':\n",
+        "        print(\"Converting weights to safetensors, please wait...\")\n",
+        "        result = subprocess.run(f\"{pyt} convert-to-safetensors.py models{slsh}{model} --output models{slsh}{model}-st\", shell=True) #convert to safetensors (Credit to oobabooga for this script as well)\n",
+        "        if result.returncode != 0:\n",
+        "            print(\"Converting failed. Please look for a safetensors model or convert model manually.\")\n",
+        "            sys.exit(\"Exiting...\")\n",
+        "        subprocess.run(f\"{osrmd} models{slsh}{model}\", shell=True)\n",
+        "        subprocess.run(f\"{osmv} models{slsh}{model}-st models{slsh}{model}\", shell=True)\n",
+        "        open(f\"models{slsh}{model}{slsh}converted-st\", 'w').close()\n",
+        "        print(\"Finished converting\")\n",
+        "        print(\"\\n\\n\")\n",
+        "    else:\n",
+        "        sys.exit(\"Can't quantize a non-safetensors model. Exiting...\")\n",
+        "\n",
+        "#create new repo if one doesn't already exist\n",
+        "if repo_exists(f\"{whoami().get('name', None)}/{modelname}-exl2\") == False:\n",
+        "    print(\"Creating model repository...\")\n",
+        "    create_repo(f\"{whoami().get('name', None)}/{modelname}-exl2\", private=True)\n",
+        "    print(f\"Created repo at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2\") #notify user of repo creation\n",
+        "\n",
+        "    #create the markdown file\n",
+        "    print(\"Writing model card...\")\n",
+        "    with open('./README.md', 'w') as file:\n",
+        "        file.write(f\"# Exl2 quants for [{modelname}](https://huggingface.co/{repo_url})\\n\\n\")\n",
+        "        file.write(\"## Automatically quantized using the auto quant from [hf-scripts](https://huggingface.co/anthonyg5005/hf-scripts)\\n\\n\")\n",
+        "        file.write(f\"Would recommend {whoami().get('name', None)} to change up this README to include more info.\\n\\n\")\n",
+        "        file.write(\"### BPW:\\n\\n\")\n",
+        "        for bpw in bpwvalue:\n",
+        "            file.write(f\"[{bpw}](https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2/tree/{bpw}bpw)\\n\\n\")\n",
+        "    print(\"Created README.md\")\n",
+        "\n",
+        "    upload_file(path_or_fileobj=\"README.md\", path_in_repo=\"README.md\", repo_id=f\"{whoami().get('name', None)}/{modelname}-exl2\", commit_message=\"Add temp README\") #upload md file\n",
+        "    print(\"Uploaded README.md to main\")\n",
+        "else:\n",
+        "    print(f\"WARNING: repo already exists at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2\")\n",
+        "\n",
+        "#start converting\n",
+        "for bpw in bpwvalue:\n",
+        "    if os.path.exists(f\"{model}-measure{slsh}measurement.json\"): # Check if measurement.json exists\n",
+        "        cmdir = False\n",
+        "        mskip = f\" -m {model}-measure{slsh}measurement.json\" #skip measurement if it exists\n",
+        "    else:\n",
+        "        cmdir = True\n",
+        "        mskip = \"\"\n",
+        "    print(f\"Starting quantization for BPW {bpw}. Please wait, may take hours\")\n",
+        "    os.makedirs(f\"{model}-exl2-{bpw}bpw-WD\", exist_ok=True) #create working directory\n",
+        "    os.makedirs(f\"{model}-exl2-{bpw}bpw\", exist_ok=True) #create compile full directory\n",
+        "    subprocess.run(f\"{oscp} models{slsh}{model}{slsh}config.json {model}-exl2-{bpw}bpw-WD\", shell=True) #copy config to working directory\n",
+        "    #more settings exist in the convert.py script, to veiw them go to docs/convert.md or https://github.com/turboderp/exllamav2/blob/master/doc/convert.md\n",
+        "    result = subprocess.run(f\"{pyt} exllamav2/convert.py -i models/{model} -o {model}-exl2-{bpw}bpw-WD -cf {model}-exl2-{bpw}bpw -b {bpw}{mskip} -ss 2048\", shell=True) #run quantization and exit if failed (Credit to turbo for his dedication to exl2)\n",
+        "    if result.returncode != 0:\n",
+        "        print(\"Quantization failed.\")\n",
+        "        sys.exit(\"Exiting...\")\n",
+        "    print(f\"Down quantizing BPW {bpw}. Starting upload\")\n",
+        "    if cmdir == True:\n",
+        "        os.makedirs(f\"{model}-measure\", exist_ok=True) #create measurement directory\n",
+        "        subprocess.run(f\"{oscp} {model}-exl2-{bpw}bpw-WD{slsh}measurement.json {model}-measure\", shell=True) #copy measurement to measure directory\n",
+        "        open(f\"{model}-measure/Delete folder when no more quants are needed from this model\", 'w').close()\n",
+        "    try:\n",
+        "        create_branch(f\"{whoami().get('name', None)}/{modelname}-exl2\", branch=f\"{bpw}bpw\") #create branch\n",
+        "    except:\n",
+        "        print(f\"Branch {bpw} already exists, trying upload...\")\n",
+        "    upload_folder(folder_path=f\"{model}-exl2-{bpw}bpw\", repo_id=f\"{whoami().get('name', None)}/{modelname}-exl2\", commit_message=f\"Add quant for BPW {bpw}\", revision=f\"{bpw}bpw\") #upload quantized model\n",
+        "    subprocess.run(f\"{osrmd} {model}-exl2-{bpw}bpw-WD\", shell=True) #remove working directory\n",
+        "    subprocess.run(f\"{osrmd} {model}-exl2-{bpw}bpw\", shell=True) #remove compile directory\n",
+        "\n",
+        "upload_file(path_or_fileobj=f\"{model}-measure{slsh}measurement.json\", path_in_repo=\"measurement.json\", repo_id=f\"{whoami().get('name', None)}/{modelname}-exl2\", commit_message=\"Add measurement.json\") #upload measurement.json to main\n",
+        "\n",
+        "print(f'''Quants available at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2\n",
+        "      \\nRepo is private, go to https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2/settings to make public if you'd like.''')\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "gpuType": "T4",
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}