File size: 4,831 Bytes
256a159 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
03/06 17:37:43 - OpenCompass - INFO - Task [my_api/siqa]
/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
03/06 17:37:51 - OpenCompass - INFO - Start inferencing [my_api/siqa]
[2024-03-06 17:37:51,870] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
0%| | 0/245 [00:00<?, ?it/s]
0%| | 0/245 [00:09<?, ?it/s]
Traceback (most recent call last):
File "/export/home/tanwentao1/project2/opencompass/opencompass/tasks/openicl_infer.py", line 153, in <module>
inferencer.run()
File "/export/home/tanwentao1/project2/opencompass/opencompass/tasks/openicl_infer.py", line 81, in run
self._inference()
File "/export/home/tanwentao1/project2/opencompass/opencompass/tasks/openicl_infer.py", line 126, in _inference
inferencer.inference(retriever,
File "/export/home/tanwentao1/project2/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 146, in inference
results = self.model.generate_from_template(
File "/export/home/tanwentao1/project2/opencompass/opencompass/models/base.py", line 165, in generate_from_template
return self.generate(inputs, max_out_len=max_out_len, **kwargs)
File "/export/home/tanwentao1/project2/opencompass/opencompass/models/my_api.py", line 79, in generate
results = list(
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
yield _result_or_cancel(fs.pop())
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
return fut.result(timeout)
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/export/home/tanwentao1/project2/opencompass/opencompass/models/my_api.py", line 151, in _generate
msg = response['choices'][0]['message']['content']
TypeError: string indices must be integers
[2024-03-06 17:38:06,766] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 269672) of binary: /export/home/tanwentao1/anaconda3/envs/opencompass/bin/python
Traceback (most recent call last):
File "/export/home/tanwentao1/anaconda3/envs/opencompass/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
run(args)
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/export/home/tanwentao1/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/export/home/tanwentao1/project2/opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-03-06_17:38:06
host : v100-4-11-71-1-176
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 269672)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
|