Repeated consecutive run failures of unrelated models on same day - (Error 28: no space left on device)

#12
by CombinHorizon - opened

is this due to an issue unrelated to these models, (maybe some temporary environment or hardware issue?)
why did that happen, could these models be restarted?

these models:
AALF/gemma-2-27b-it-SimPO-37K
AALF/gemma-2-27b-it-SimPO-37K
abacusai/bigstral-12b-32k
abacusai/bigyi-15b
abacusai/Slerp-CM-mist-dpo
anthracite-org/magnum-v2.5-12b-kto
BAAI/Gemma2-9B-IT-Simpo-Infinity-Preference
bunnycore/HyperLlama-3.1-8B
byroneverson/gemma-2-27b-it-abliterated
EpistemeAI2/Fireball-Alpaca-Llama3.1.06-8B-Philos
jpacifico/Chocolatine-14B-Instruct-DPO-v1.2
migtissera/Tess-3-Mistral-Nemo-12B
migtissera/Tess-v2.5-Gemma-2-27B-alpha
monsterapi/Llama-3_1-8B-Instruct-orca-ORPO
nbeerbower/mistral-nemo-wissenschaft-12B
TheDrummer/Big-Tiger-Gemma-27B-v1

Request files:
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/AALF/gemma-2-27b-it-SimPO-37K_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/AALF/gemma-2-27b-it-SimPO-37K_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/abacusai/bigstral-12b-32k_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/abacusai/bigyi-15b_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/abacusai/Slerp-CM-mist-dpo_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/anthracite-org/magnum-v2.5-12b-kto_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/BAAI/Gemma2-9B-IT-Simpo-Infinity-Preference_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/bunnycore/HyperLlama-3.1-8B_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/byroneverson/gemma-2-27b-it-abliterated_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/EpistemeAI2/Fireball-Alpaca-Llama3.1.06-8B-Philos_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/jpacifico/Chocolatine-14B-Instruct-DPO-v1.2_eval_request_False_float16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/migtissera/Tess-3-Mistral-Nemo-12B_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/migtissera/Tess-v2.5-Gemma-2-27B-alpha_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/monsterapi/Llama-3_1-8B-Instruct-orca-ORPO_eval_request_False_bfloat16_Adapter.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/nbeerbower/mistral-nemo-wissenschaft-12B_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests/blob/main/TheDrummer/Big-Tiger-Gemma-27B-v1_eval_request_False_bfloat16_Original.json

Sign up or log in to comment