LLM Foundry Updates 06-01-2023
#41
by
abhi-mosaic
- opened
** Duplicate of https://huggingface.co/mosaicml/mpt-7b/discussions/47 **
This PR adds updates from the LLM Foundry repo as of 06/01/2023.
These include:
device_mapsupport for multiple GPUs- faster inference thanks to a refactor of the KV cacheing
- bugfix for returning the last hidden_state
- support for
output_attentionswhen usingattn_impl: torch - a
requirements.txtfile to make it easier to know what you need to install for MPT - updated README instructions for fast GPU initialization
abhi-mosaic
changed pull request status to
open
Tested with LLM Foundry with python scripts/inference/hf_generate.py -n mosaicml/mpt-7b-instruct --revision pr/41 --device_map auto
abhi-mosaic
changed pull request status to
merged