add local version of script

tested. Modified version of auto exl2 upload. also some small changes.

README.md

@@ -16,14 +16,14 @@ Feel free to send in PRs or use this code however you'd like.\
 
 - [Manage branches (create/delete)](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/manage%20branches.py)
 
-- [Auto EXL2 upload](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/auto-exl2-upload/auto-exl2-upload.zip?download=true)
+- [Auto EXL2 HF upload](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/auto-exl2-upload/auto-exl2-upload.zip?download=true)
 
-- [EXL2 Single Quant V3](https://colab.research.google.com/drive/1Vc7d6JU3Z35OVHmtuMuhT830THJnzNfS?usp=sharing) **(COLAB)**
-
-- [EXL2 Local Quant - Windows](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/exl2-windows-local/exl2-windows-local.zip?download=true)
+- [EXL2 Local Quants](https://huggingface.co/Anthonyg5005/hf-scripts/resolve/main/exl2-multi-quant-local/exl2-multi-quant-local.zip?download=true)
 
 - [Upload folder to HF](https://huggingface.co/Anthonyg5005/hf-scripts/blob/main/upload%20folder%20to%20repo.py)
 
+- [EXL2 Single Quant V3](https://colab.research.google.com/drive/1Vc7d6JU3Z35OVHmtuMuhT830THJnzNfS?usp=sharing) **(COLAB)**
+
 ## work in progress/not tested ([unfinished](https://huggingface.co/Anthonyg5005/hf-scripts/tree/unfinished) branch)
 
 - EXL2 Multi Quant local
@@ -31,7 +31,7 @@ Feel free to send in PRs or use this code however you'd like.\
 
 ## other recommended stuff
 
-- [Exllama Discord server](https://discord.gg/NSFwVuCjRq) Free Exl2 quantizing bot sponsored by The Bloke and managed by kaltcit.
+- [Exllama Discord server](https://discord.gg/NSFwVuCjRq) Free Exl2 quantizing bot sponsored by The Bloke and Lambda Labs, managed by Kaltcit.
   - existing quants under the HF account [@blockblockblock](https://huggingface.co/blockblockblock)
 
 - [Download models (download HF Hub models) [Oobabooga]](https://github.com/oobabooga/text-generation-webui/blob/main/download-model.py)
@@ -41,11 +41,11 @@ Feel free to send in PRs or use this code however you'd like.\
 - Manage branches
   - Run script and follow prompts. You will be required to be logged in to HF Hub. If you are not logged in, you will need a WRITE token. You can get one in your [HuggingFace settings](https://huggingface.co/settings/tokens). Colab and Kaggle secret keys are supported.
 
-- Auto EXL2 upload
-  - This script is designed to automate the process of quantizing models to EXL2 and uploading them to the HF Hub as seperate branches. This is both available to run on Windows and Linux.
+- Auto EXL2 HF upload
+  - This script is designed to automate the process of quantizing models to EXL2 and uploading them to the HF Hub as seperate branches. This is both available to run on Windows and Linux. You will be required to be logged in to HF Hub. If you are not logged in, you will need a WRITE token.
 
-- EXL2 Local Quant Windows
-  - Easily creates environment to quantize models to exl2 using Windows to your local machine. Replacing soon.
+- EXL2 Local Quants
+  - Easily creates environment to quantize models to exl2 to your local machine. Supports both Windows and Linux.
 
 - Upload folder to repo
   - Uploads user specified folder to specified repo, can create private repos too. Not the same as git commit and push, instead uploads any additional files. This is more to be modified to your needs then used by itself.

auto-exl2-upload/INSTRUCTIONS.txt

@@ -10,7 +10,7 @@ https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows&targ
 Haven't done much testing but for Windows, Visual Studio with desktop development for C++ might be required. I've gotten cl.exe errors on a previous install
 
 
-This may work with AMD cards but only on linux. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
+This may work with AMD cards but only on linux and possibly WSL2. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
 
 
 

auto-exl2-upload/auto-exl2-upload.zip

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb52c9577ce88784c0831237f91a9d3edcdcafb3c85ec94b8dac85e036daf8a4
-size 6692
+oid sha256:2696d6221f7867d2737e95ec1127df4364a22b3464a018baaa9ddd95df87e8a4
+size 6726

auto-exl2-upload/exl2-quant.py

@@ -1,4 +1,4 @@
-#usually it's on the inside that counts, not this time. This script is a mess, but it works.
+#usually it's what is on the inside that counts, not this time. This script is a mess, but at least it works.
 #import required modules
 from huggingface_hub import login, get_token, whoami, repo_exists, file_exists, upload_folder, create_repo, upload_file, create_branch
 import os
@@ -188,3 +188,11 @@ if file_exists(f"{whoami().get('name', None)}/{modelname}-exl2", "measurement.js
     
 print(f'''Quants available at https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2
       \nRepo is private, go to https://huggingface.co/{whoami().get('name', None)}/{modelname}-exl2/settings to make public if you'd like.''')
+
+if tfound == 'false':
+    print(f'''
+              You are now logged in as {whoami().get('fullname', None)}.
+          
+          To logout, use the hf command line interface 'huggingface-cli logout'
+               To view your active account, use 'huggingface-cli whoami'
+          ''')

auto-exl2-upload/linux-setup.sh

@@ -55,9 +55,10 @@ venv/bin/python -m pip install ./exllamav2
 # create start-quant.sh
 echo "#!/bin/bash" > start-quant.sh
 echo "venv/bin/python exl2-quant.py" >> start-quant.sh
-echo "read -p \"Press enter to continue\""
+echo "read -p \"Press enter to continue\"" >> start-quant.sh
+echo "exit" >> start-quant.sh
 chmod +x start-quant.sh
 echo "If you use ctrl+c to stop, you may need to also use 'pkill python' to stop running scripts."
 echo "Environment setup complete. run start-quant.sh to start the quantization process."
 read -p "Press enter to exit"
-exit
+exit

exl2-multi-quant-local/INSTRUCTIONS.txt

@@ -0,0 +1,38 @@
+For NVIDIA cards install the CUDA toolkit
+
+Nvidia Maxwell or higher
+https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64
+
+Nvidia Kepler or higher
+https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows&target_arch=x86_64
+
+
+Haven't done much testing but for Windows, Visual Studio with desktop development for C++ might be required. I've gotten cl.exe errors on a previous install
+
+
+This may work with AMD cards but only on linux and possibly WSL2. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
+
+
+
+First setup your environment by using either windows.bat or linux.sh. If something fails during setup, then every file and folder except for windows.bat, linux.sh, and exl2-quant.py should be deleted then try again.
+
+After setup is complete then you'll have a file called start-quant. Use this to run the quant script.
+
+Make sure that your storage space is 3x the amount of the model's size plus 1 more time per quant. To mesure this, take the number of billion parameters and mutliply by two, afterwards mutliply by 3 and that's the recommended storage. There's a chance you may get away with 2.5x the size as well.
+Make sure to also have a lot of RAM depending on the model.
+
+If you close the terminal or the terminal crashes, check the last BPW it was on and enter the remaining quants you wanted. It should be able to pick up where it left off. Don't type the finished BPW as it will start from the beginning. You may also use ctrl + c pause at any time during the quant process.
+
+Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+
+
+Credit to turboderp for creating exllamav2 and the exl2 quantization method.
+https://github.com/turboderp
+
+Credit to oobabooga the original download and safetensors scripts.
+https://github.com/oobabooga
+
+Credit to Lucain Pouget for maintaining huggingface-hub.
+https://github.com/Wauplin
+
+Only tested with CUDA 12.1 on Windows 11 and half-tested Linux through WSL2 but I don't have enough RAM to fully test but quantization did start.

exl2-multi-quant-local/exl2-multi-quant-local.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8ab35ce67e701370b10168a4d745333e2cb70e098c5c469ea518e07bb57fe9d
+size 5883

exl2-multi-quant-local/exl2-quant.py

@@ -0,0 +1,144 @@
+#usually it's what is on the inside that counts, not this time. This script is a mess, but at least it works.
+#import required modules
+from huggingface_hub import login, get_token, whoami, repo_exists
+import os
+import sys
+import subprocess
+import glob
+import time
+
+#define os differences
+oname = os.name
+if oname == 'nt':
+    osclear = 'cls'
+    osmv = 'move'
+    osrmd = 'rmdir /s /q'
+    oscp = 'copy'
+    pyt = 'venv\\scripts\\python.exe'
+    slsh = '\\'
+elif oname == 'posix':
+    osclear = 'clear'
+    osmv = 'mv'
+    osrmd = 'rm -r'
+    oscp = 'cp'
+    pyt = './venv/bin/python'
+    slsh = '/'
+else:
+    sys.exit('This script is not compatible with your machine.')
+def clear_screen():
+    os.system(osclear)
+
+#get token
+if os.environ.get('KAGGLE_KERNEL_RUN_TYPE', None) is not None: #check if user in kaggle
+    from kaggle_secrets import UserSecretsClient # type: ignore
+    from kaggle_web_client import BackendError # type: ignore
+    try:
+        login(UserSecretsClient().get_secret("HF_TOKEN")) #login if token secret found
+    except BackendError: 
+        print('''
+            When using Kaggle, make sure to use the secret key HF_TOKEN with a 'WRITE' token.
+                   This will prevent the need to login every time you run the script.
+                   Set your secrets with the secrets add-on on the top of the screen.
+             ''')
+if get_token() is not None:
+    #if the token is found then log in:
+    login(get_token())
+    tfound = "true" 
+else:
+    #if the token is not found then prompt user to provide it:
+    tfound = "false"
+    try:
+        login(input("API token not detected. Enter your HuggingFace token (empty to skip): "))
+    except:
+        print("Skipping login... (Unable to access private or gated models)")
+        tfound = "false but skipped" #doesn't matter what this is, only false is used
+        time.sleep(3)
+
+clear_screen()
+
+#get original model repo url
+repo_url = input("Enter unquantized model repository (User/Repo): ")
+
+#look for repo
+if repo_exists(repo_url) == False:
+    print(f"Model repo doesn't exist at https://huggingface.co/{repo_url}")
+    sys.exit("Exiting...")
+model = repo_url.replace("/", "_")
+modelname = repo_url.split("/")[1]
+clear_screen()
+
+#ask for number of quants
+qmount = int(input("Enter the number of quants you want to create: "))
+qmount += 1
+clear_screen()
+
+#save bpw values
+print(f"Type the BPW for the following {qmount - 1} quants. Recommend staying over 2.4 BPW. Use the vram calculator to find the best BPW values: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator")
+qnum = {}
+for i in range(1, qmount):
+    qnum[f"bpw{i}"] = float(input(f"Enter BPW for quant {i} (2.00-8.00): ")) #convert input to float for proper sorting
+clear_screen()
+
+#collect all values in a list for sorting
+bpwvalue = list(qnum.values())
+
+#sort the list from smallest to largest
+bpwvalue.sort()
+
+if not os.path.exists(f"models{slsh}{model}{slsh}converted-st"): #check if model was converted to safetensors, skip download if it was
+    result = subprocess.run(f"{pyt} download-model.py {repo_url}", shell=True) #download model from hf (Credit to oobabooga for this script)
+    if result.returncode != 0:
+        print("Download failed.")
+        sys.exit("Exiting...")
+    clear_screen()
+
+if not glob.glob(f"models/{model}/*.safetensors"): #check if safetensors model exists
+    convertst = input("Couldn't find safetensors model, do you want to convert to safetensors? (y/n): ")
+    while convertst != 'y' and convertst != 'n':
+        convertst = input("Please enter 'y' or 'n': ")
+    if convertst == 'y':
+        print("Converting weights to safetensors, please wait...")
+        result = subprocess.run(f"{pyt} convert-to-safetensors.py models{slsh}{model} --output models{slsh}{model}-st", shell=True) #convert to safetensors (Credit to oobabooga for this script as well)
+        if result.returncode != 0:
+            print("Converting failed. Please look for a safetensors model or convert model manually.")
+            sys.exit("Exiting...")
+        subprocess.run(f"{osrmd} models{slsh}{model}", shell=True)
+        subprocess.run(f"{osmv} models{slsh}{model}-st models{slsh}{model}", shell=True)
+        open(f"models{slsh}{model}{slsh}converted-st", 'w').close()
+        print("Finished converting")
+    else:
+        sys.exit("Can't quantize a non-safetensors model. Exiting...")
+clear_screen()
+
+#start converting
+for bpw in bpwvalue:
+    if os.path.exists(f"{model}-measure{slsh}measurement.json"): # Check if measurement.json exists
+        cmdir = False
+        mskip = f" -m {model}-measure{slsh}measurement.json" #skip measurement if it exists
+    else:
+        cmdir = True
+        mskip = ""
+    print(f"Starting quantization for BPW {bpw}")
+    os.makedirs(f"{model}-exl2-{bpw}bpw-WD", exist_ok=True) #create working directory
+    os.makedirs(f"{modelname}-exl2-quants{slsh}{modelname}-exl2-{bpw}bpw", exist_ok=True) #create compile full directory
+    subprocess.run(f"{oscp} models{slsh}{model}{slsh}config.json {model}-exl2-{bpw}bpw-WD", shell=True) #copy config to working directory
+    #more settings exist in the convert.py script, to veiw them go to docs/convert.md or https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
+    result = subprocess.run(f"{pyt} exllamav2/convert.py -i models/{model} -o {model}-exl2-{bpw}bpw-WD -cf {modelname}-exl2-quants{slsh}{modelname}-exl2-{bpw}bpw -b {bpw}{mskip}", shell=True) #run quantization and exit if failed (Credit to turbo for his dedication to exl2)
+    if result.returncode != 0:
+        print("Quantization failed.")
+        sys.exit("Exiting...")
+    if cmdir == True:
+        os.makedirs(f"{model}-measure", exist_ok=True) #create measurement directory
+        subprocess.run(f"{oscp} {model}-exl2-{bpw}bpw-WD{slsh}measurement.json {model}-measure", shell=True) #copy measurement to measure directory
+        open(f"{model}-measure/Delete folder when no more quants are needed from this model", 'w').close()
+    subprocess.run(f"{osrmd} {model}-exl2-{bpw}bpw-WD", shell=True) #remove working directory
+
+if tfound == 'false':
+    print(f'''
+              You are now logged in as {whoami().get('fullname', None)}.
+          
+          To logout, use the hf command line interface 'huggingface-cli logout'
+               To view your active account, use 'huggingface-cli whoami'
+          ''')
+    
+print("Finished quantizing. Exiting...")

exl2-multi-quant-local/linux-setup.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+
+# converted from windows-setup.bat by github copilot
+
+# check if "venv" subdirectory exists, if not, create one
+if [ ! -d "venv" ]; then
+    python -m venv venv
+else
+    echo "venv directory already exists. If something is broken, delete everything but exl2-quant.py and run this script again."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# ask if the user has git installed
+read -p "Do you have git and wget installed? (y/n) " gitwget
+
+if [ "$gitwget" = "y" ]; then
+    echo "Setting up environment"
+else
+    echo "Please install git and wget before running this script."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8. If ROCm, install pytorch for ROCm 5.7
+read -p "Please enter your GPU compute version, CUDA 11/12 or AMD ROCm (11, 12, rocm): " pytorch_version
+
+if [ "$pytorch_version" = "11" ]; then
+    echo "Installing PyTorch for CUDA 11.8"
+    venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cu118
+elif [ "$pytorch_version" = "12" ]; then
+    echo "Installing PyTorch for CUDA 12.1"
+    venv/bin/python -m pip install torch
+elif [ "$pytorch_version" = "rocm" ]; then
+    echo "Installing PyTorch for AMD ROCm 5.7"
+    venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/rocm5.7
+else
+    echo "Invalid compute version. Please enter 11, 12, or rocm."
+    read -p "Press enter to continue"
+    exit
+fi
+
+# download stuff
+echo "Downloading files"
+git clone https://github.com/turboderp/exllamav2
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/convert-to-safetensors.py
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/download-model.py
+
+echo "Installing pip packages"
+
+venv/bin/python -m pip install -r exllamav2/requirements.txt
+venv/bin/python -m pip install huggingface-hub transformers accelerate
+venv/bin/python -m pip install ./exllamav2
+
+# create start-quant.sh
+echo "#!/bin/bash" > start-quant.sh
+echo "venv/bin/python exl2-quant.py" >> start-quant.sh
+echo "read -p \"Press enter to continue\"" >> start-quant.sh
+echo "exit" >> start-quant.sh
+chmod +x start-quant.sh
+echo "If you use ctrl+c to stop, you may need to also use 'pkill python' to stop running scripts."
+echo "Environment setup complete. run start-quant.sh to start the quantization process."
+read -p "Press enter to exit"
+exit

exl2-multi-quant-local/windows-setup.bat

@@ -0,0 +1,63 @@
+@echo off
+
+setlocal
+
+REM check if "venv" subdirectory exists, if not, create one
+if not exist "venv\" (
+    python -m venv venv
+) else (
+    echo venv directory already exists. If something is broken, delete everything but exl2-quant.py and run this script again.
+    pause
+    exit
+)
+
+REM ask if the user has git installed
+set /p gitwget="Do you have git and wget installed? (y/n) "
+
+if "%gitwget%"=="y" (
+    echo "Setting up environment"
+) else (
+    echo Please install git and wget before running this script.
+    echo winget install wget
+    echo winget install git
+    pause
+    exit
+)
+
+REM if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8
+echo CUDA path: %CUDA_HOME%
+set /p cuda_version="Please enter your CUDA version (11 or 12): "
+
+if "%cuda_version%"=="11" (
+    echo Installing PyTorch for CUDA 11.8...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu118
+) else if "%cuda_version%"=="12" (
+    echo Installing PyTorch for CUDA 12.1...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu121
+) else (
+    echo Invalid CUDA version. Please enter 11 or 12.
+    pause
+    exit
+)
+
+REM download stuff
+echo Downloading files...
+git clone https://github.com/turboderp/exllamav2
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/convert-to-safetensors.py
+wget https://raw.githubusercontent.com/oobabooga/text-generation-webui/main/download-model.py
+
+echo Installing pip packages...
+
+venv\scripts\python.exe -m pip install -r exllamav2/requirements.txt
+venv\scripts\python.exe -m pip install huggingface-hub transformers accelerate
+venv\scripts\python.exe -m pip install .\exllamav2
+
+REM create start-quant-windows.bat
+echo @echo off > start-quant.bat
+echo venv\scripts\python.exe exl2-quant.py >> start-quant.bat
+echo REM tada sound for fun >> start-quant.bat
+echo powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync(); >> start-quant.bat
+echo pause >> start-quant.bat
+powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync();
+echo Environment setup complete. run start-quant.bat to start the quantization process.
+pause