
# RWKV-4 PilePlus

## Model Description

RWKV-4-pile models finetuning on [RedPajama + some of Pile v2 = 1.7T tokens]. Updated with 2020+2021+2022 data, and better at all European languages.

Although some of these are intermedia checkpoints (XXXGtokens means finetuned for XXXG tokens), you can already use them because I am finetuning from Pile models (instead of retraining).

Note: not instruct tuned yet, and recommended to replace vanilla Pile models.

7B and 14B coming soon.

See https://github.com/BlinkDL/RWKV-LM for details.

Use https://github.com/BlinkDL/ChatRWKV to run it.

{"language": ["en"], "license": "apache-2.0", "tags": ["pytorch", "text-generation", "causal-lm", "rwkv"], "datasets": ["EleutherAI/pile", "togethercomputer/RedPajama-Data-1T"]}
pytorch text-generation causal-lm rwkv en dataset:EleutherAI/pile dataset:togethercomputer/RedPajama-Data-1T license:apache-2.0 has_space region:us
TAGS
#pytorch #text-generation #causal-lm #rwkv #en #dataset-EleutherAI/pile #dataset-togethercomputer/RedPajama-Data-1T #license-apache-2.0 #has_space #region-us 


# RWKV-4 PilePlus

## Model Description

RWKV-4-pile models finetuning on [RedPajama + some of Pile v2 = 1.7T tokens]. Updated with 2020+2021+2022 data, and better at all European languages.

Although some of these are intermedia checkpoints (XXXGtokens means finetuned for XXXG tokens), you can already use them because I am finetuning from Pile models (instead of retraining).

Note: not instruct tuned yet, and recommended to replace vanilla Pile models.

7B and 14B coming soon.

See URL for details.

Use URL to run it.

# RWKV-4 PilePlus ## Model Description

RWKV-4-pile models finetuning on [RedPajama + some of Pile v2 = 1.7T tokens]. Updated with 2020+2021+2022 data, and better at all European languages.

Although some of these are intermedia checkpoints (XXXGtokens means finetuned for XXXG tokens), you can already use them because I am finetuning from Pile models (instead of retraining).

Note: not instruct tuned yet, and recommended to replace vanilla Pile models.

7B and 14B coming soon.

See URL for details.

Use URL to run it.
TAGS
#pytorch #text-generation #causal-lm #rwkv #en #dataset-EleutherAI/pile #dataset-togethercomputer/RedPajama-Data-1T #license-apache-2.0 #has_space #region-us 
 # RWKV-4 PilePlus ## Model Description

RWKV-4-pile models finetuning on [RedPajama + some of Pile v2 = 1.7T tokens]. Updated with 2020+2021+2022 data, and better at all European languages.

Although some of these are intermedia checkpoints (XXXGtokens means finetuned for XXXG tokens), you can already use them because I am finetuning from Pile models (instead of retraining).

Note: not instruct tuned yet, and recommended to replace vanilla Pile models.

7B and 14B coming soon.

See URL for details.

Use URL to run it.