Spaces:

hallucinations-leaderboard

/

leaderboard

like 123

Running on CPU Upgrade

Thanks! Looking into this.
Update: the harness is able to handle it without issues, so I think I can upload the leaderboard code to get rid of the issue!

%cd /content/lm-evaluation-harness
!lm_eval --model hf --model_args pretrained=AI-Sweden-Models/gpt-sw3-20b-instruct,revision=main --tasks nq_open --device cuda --batch_size 1 --num_fewshot 8

2024-02-03:07:55:56,177 INFO     [utils.py:148] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-02-03:07:55:56,177 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2024-02-03:07:55:56,352 INFO     [config.py:58] PyTorch version 2.1.0+cu121 available.
2024-02-03:07:55:56,353 INFO     [config.py:95] TensorFlow version 2.15.0 available.
2024-02-03:07:55:56,354 INFO     [config.py:108] JAX version 0.4.23 available.
2024-02-03 07:55:56.917234: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-03 07:55:56.917328: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-03 07:55:56.918882: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-03 07:55:58.092248: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-02-03:07:56:00,083 INFO     [__main__.py:162] Verbosity set to INFO
2024-02-03:07:56:00,083 INFO     [__init__.py:358] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2024-02-03:07:56:04,311 INFO     [__main__.py:238] Selected Tasks: ['nq_open']
2024-02-03:07:56:04,311 INFO     [__main__.py:239] Loading selected tasks...
2024-02-03:07:56:04,366 INFO     [huggingface.py:148] Using device 'cuda'
config.json: 100% 995/995 [00:00<00:00, 5.92MB/s]
model.safetensors.index.json: 100% 48.8k/48.8k [00:00<00:00, 4.96MB/s]
Downloading shards:   0% 0/9 [00:00<?, ?it/s]
[..]

@Ekgren can you try again?

Now it works! Thank you