beomi commited on
Commit
cfac31c
1 Parent(s): 192ae02

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +219 -0
README.md ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ko
4
+ - en
5
+ license: other
6
+ library_name: transformers
7
+ license_name: gemma-terms-of-use
8
+ license_link: https://ai.google.dev/gemma/terms
9
+ pipeline_tag: text-generation
10
+ tags:
11
+ - pytorch
12
+ ---
13
+
14
+ # Gemma-Ko
15
+
16
+ > Update @ 2024.03.26: First release of Gemma-Ko 2B model
17
+
18
+ **Original Gemma Model Page**: [Gemma](https://ai.google.dev/gemma/docs)
19
+
20
+ This model card corresponds to the 7B base version of the **Gemma-Ko** model.
21
+
22
+ **Resources and Technical Documentation**:
23
+
24
+ * [Original Google's Gemma-2B](https://huggingface.co/google/gemma-2b)
25
+ * [Training Code @ Github: Gemma-EasyLM](https://github.com/Beomi/Gemma-EasyLM)
26
+
27
+ **Terms of Use**: [Terms](https://www.kaggle.com/models/google/gemma/license/consent)
28
+
29
+ **Citation**
30
+
31
+ ```bibtex
32
+ @misc {gemma_ko_7b,
33
+ author = { {Junbum Lee, Taekyoon Choi} },
34
+ title = { gemma-ko-7b },
35
+ year = 2024,
36
+ url = { https://huggingface.co/beomi/gemma-ko-7b },
37
+ doi = { 10.57967/hf/1859 },
38
+ publisher = { Hugging Face }
39
+ }
40
+ ```
41
+
42
+ **Model Developers**: Junbum Lee (Beomi) & Taekyoon Choi (Taekyoon)
43
+
44
+ ## Model Information
45
+
46
+ Summary description and brief definition of inputs and outputs.
47
+
48
+ ### Description
49
+
50
+ Gemma is a family of lightweight, state-of-the-art open models from Google,
51
+ built from the same research and technology used to create the Gemini models.
52
+ They are text-to-text, decoder-only large language models, available in English,
53
+ with open weights, pre-trained variants, and instruction-tuned variants. Gemma
54
+ models are well-suited for a variety of text generation tasks, including
55
+ question answering, summarization, and reasoning. Their relatively small size
56
+ makes it possible to deploy them in environments with limited resources such as
57
+ a laptop, desktop or your own cloud infrastructure, democratizing access to
58
+ state of the art AI models and helping foster innovation for everyone.
59
+
60
+ ### Usage
61
+
62
+ Below we share some code snippets on how to get quickly started with running the model. First make sure to `pip install -U transformers`, then copy the snippet from the section that is relevant for your usecase.
63
+
64
+ #### Running the model on a CPU
65
+
66
+ ```python
67
+ from transformers import AutoTokenizer, AutoModelForCausalLM
68
+
69
+ tokenizer = AutoTokenizer.from_pretrained("beomi/gemma-ko-2b")
70
+ model = AutoModelForCausalLM.from_pretrained("beomi/gemma-ko-2b")
71
+
72
+ input_text = "머신러닝과 딥러닝의 차이는"
73
+ input_ids = tokenizer(input_text, return_tensors="pt")
74
+
75
+ outputs = model.generate(**input_ids)
76
+ print(tokenizer.decode(outputs[0]))
77
+ ```
78
+
79
+
80
+ #### Running the model on a single / multi GPU
81
+
82
+ ```python
83
+ # pip install accelerate
84
+ from transformers import AutoTokenizer, AutoModelForCausalLM
85
+
86
+ tokenizer = AutoTokenizer.from_pretrained("beomi/gemma-ko-2b")
87
+ model = AutoModelForCausalLM.from_pretrained("beomi/gemma-ko-2b", device_map="auto")
88
+
89
+ input_text = "머신러닝과 딥러닝의 차이는"
90
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
91
+
92
+ outputs = model.generate(**input_ids)
93
+ print(tokenizer.decode(outputs[0]))
94
+ ```
95
+
96
+ #### Other optimizations
97
+
98
+ * _Flash Attention 2_
99
+
100
+ First make sure to install `flash-attn` in your environment `pip install flash-attn`
101
+
102
+ ```diff
103
+ model = AutoModelForCausalLM.from_pretrained(
104
+ "beomi/gemma-ko-2b",
105
+ torch_dtype=torch.float16,
106
+ + attn_implementation="flash_attention_2"
107
+ ).to(0)
108
+ ```
109
+
110
+ ### Inputs and outputs
111
+
112
+ * **Input:** Text string, such as a question, a prompt, or a document to be
113
+ summarized.
114
+ * **Output:** Generated Korean/English-language text in response to the input, such
115
+ as an answer to a question, or a summary of a document.
116
+
117
+ ## Implementation Information
118
+
119
+ Details about the model internals.
120
+
121
+ ### Software
122
+
123
+ Training was done using [beomi/Gemma-EasyLM](https://github.com/Beomi/Gemma-EasyLM).
124
+
125
+
126
+ ## Evaluation
127
+
128
+ Model evaluation metrics and results.
129
+
130
+ ### Benchmark Results
131
+
132
+ TBD
133
+
134
+ ## Usage and Limitations
135
+
136
+ These models have certain limitations that users should be aware of.
137
+
138
+ ### Intended Usage
139
+
140
+ Open Large Language Models (LLMs) have a wide range of applications across
141
+ various industries and domains. The following list of potential uses is not
142
+ comprehensive. The purpose of this list is to provide contextual information
143
+ about the possible use-cases that the model creators considered as part of model
144
+ training and development.
145
+
146
+ * Content Creation and Communication
147
+ * Text Generation: These models can be used to generate creative text formats
148
+ such as poems, scripts, code, marketing copy, and email drafts.
149
+ * Research and Education
150
+ * Natural Language Processing (NLP) Research: These models can serve as a
151
+ foundation for researchers to experiment with NLP techniques, develop
152
+ algorithms, and contribute to the advancement of the field.
153
+ * Language Learning Tools: Support interactive language learning experiences,
154
+ aiding in grammar correction or providing writing practice.
155
+ * Knowledge Exploration: Assist researchers in exploring large bodies of text
156
+ by generating summaries or answering questions about specific topics.
157
+
158
+ ### Limitations
159
+
160
+ * Training Data
161
+ * The quality and diversity of the training data significantly influence the
162
+ model's capabilities. Biases or gaps in the training data can lead to
163
+ limitations in the model's responses.
164
+ * The scope of the training dataset determines the subject areas the model can
165
+ handle effectively.
166
+ * Context and Task Complexity
167
+ * LLMs are better at tasks that can be framed with clear prompts and
168
+ instructions. Open-ended or highly complex tasks might be challenging.
169
+ * A model's performance can be influenced by the amount of context provided
170
+ (longer context generally leads to better outputs, up to a certain point).
171
+ * Language Ambiguity and Nuance
172
+ * Natural language is inherently complex. LLMs might struggle to grasp subtle
173
+ nuances, sarcasm, or figurative language.
174
+ * Factual Accuracy
175
+ * LLMs generate responses based on information they learned from their
176
+ training datasets, but they are not knowledge bases. They may generate
177
+ incorrect or outdated factual statements.
178
+ * Common Sense
179
+ * LLMs rely on statistical patterns in language. They might lack the ability
180
+ to apply common sense reasoning in certain situations.
181
+
182
+ ### Ethical Considerations and Risks
183
+
184
+ The development of large language models (LLMs) raises several ethical concerns.
185
+ In creating an open model, we have carefully considered the following:
186
+
187
+ * Bias and Fairness
188
+ * LLMs trained on large-scale, real-world text data can reflect socio-cultural
189
+ biases embedded in the training material. These models underwent careful
190
+ scrutiny, input data pre-processing described and posterior evaluations
191
+ reported in this card.
192
+ * Misinformation and Misuse
193
+ * LLMs can be misused to generate text that is false, misleading, or harmful.
194
+ * Guidelines are provided for responsible use with the model, see the
195
+ [Responsible Generative AI Toolkit](http://ai.google.dev/gemma/responsible).
196
+ * Transparency and Accountability:
197
+ * This model card summarizes details on the models' architecture,
198
+ capabilities, limitations, and evaluation processes.
199
+ * A responsibly developed open model offers the opportunity to share
200
+ innovation by making LLM technology accessible to developers and researchers
201
+ across the AI ecosystem.
202
+
203
+ Risks identified and mitigations:
204
+
205
+ * Perpetuation of biases: It's encouraged to perform continuous monitoring
206
+ (using evaluation metrics, human review) and the exploration of de-biasing
207
+ techniques during model training, fine-tuning, and other use cases.
208
+ * Generation of harmful content: Mechanisms and guidelines for content safety
209
+ are essential. Developers are encouraged to exercise caution and implement
210
+ appropriate content safety safeguards based on their specific product policies
211
+ and application use cases.
212
+ * Misuse for malicious purposes: Technical limitations and developer and
213
+ end-user education can help mitigate against malicious applications of LLMs.
214
+ Educational resources and reporting mechanisms for users to flag misuse are
215
+ provided. Prohibited uses of Gemma models are outlined in the
216
+ [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy).
217
+ * Privacy violations: Models were trained on data filtered for removal of PII
218
+ (Personally Identifiable Information). Developers are encouraged to adhere to
219
+ privacy regulations with privacy-preserving techniques.