--- license: mit language: - ja - en --- # Sarashina2-7B This repository provides large language models trained by [SB Intuitions](https://www.sbintuitions.co.jp/). ## How to use ``` import torch from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2-7b", torch_dtype=torch.bfloat16) tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-7b", use_fast=False) generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device_map="auto") set_seed(123) text = generator( "おはようございます、今日の天気は", max_length=30, do_sample=True, pad_token_id=tokenizer.pad_token_id, num_return_sequences=3, ) for t in text: print(t) # These examples are generated by sarashina2-7b parameters model # {'generated_text': 'おはようございます、今日の天気は晴れです。ちょっと風が強い。\n昨日は、久しぶりにゆっくりとしていました。\n2週間位間があいてしまったかも、でもその間に'} # {'generated_text': 'おはようございます、今日の天気は曇。朝は曇っていてどんよりしていましたね。昼からは晴れそうですが。気温は徐々に上昇しています。昨日は春らしい陽気でした。'} # {'generated_text': 'おはようございます、今日の天気はくもり、少し寒気がします。 この土日に、家族で一泊二日で旅行に行ってきました。といっても、100キロ'} ``` ## Configuration | Parameters | Vocab size | Trainning tokens | Architecture | Position type | Layers | Hidden dim | Attention heads | | :-----: | :-----------: | :-------------: | :------------ | :-----------: | :----: | :--------: | :-------------: | | [7B](https://huggingface.co/sbintuitions/sarashina2-7b) | 102400 | 2.1T | Llama2 | RoPE | 32 | 4096 | 32 | | [13B](https://huggingface.co/sbintuitions/sarashina2-13b) | 102400 | 2.1T | Llama2 | RoPE | 40 | 5120 | 40 | | 70B (TBA)| | | | | | | ## Training Corpus For our Japanese training data, we used a Japanese portion of the [Common Crawl corpus](https://commoncrawl.org/), which is the largest Web corpus, as our training dataset. To clean the training corpus, we used [CCNet](https://github.com/facebookresearch/cc_net) and [HojiChar](https://github.com/HojiChar/HojiChar). After cleaning, our Japanese training data contains about 1T tokens. For our English training data, we extracted English documents from [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B) but we removed books3 corpus due to copyright infringement. ## Tokenization We use a [sentencepiece](https://github.com/google/sentencepiece) tokenizer with a unigram language model and byte-fallback. We do not apply pre-tokenization with Japanese tokenizer. Thus, a user may directly feed raw sentences into the tokenizer. ## Ethical Considerations and Limitations Sarashina2 has not been tuned to follow an instruction yet. Therefore, sarashina2 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs. Before using sarashina2, we would like developers to tune models based on human preferences and safety considerations. ## License [MIT License](https://huggingface.co/sbintuitions/sarashina2-7b/blob/main/LICENSE)