library_name: transformers
tags:
- nlp
- phi3
license: mit
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
I recommend to use this improved model -> devngho/phi-3-mini-4k-back-to-base!
Model Card for devngho/phi-3-mini-4k-base
Finetuned from microsoft/Phi-3-mini-4k-instruct on fineweb-edu (about 150,000 samples).
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). ⚡
Model Details
- Developed by: devngho
- Language(s) (NLP): en
- License: mit
- Parent Model: microsoft/Phi-3-mini-4k-instruct
Samples
- max_tokens: 256
- <s> token is omitted
- temperature: 0
Sample 1
Prompt:
Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
This model(devngho/phi-3-mini-4k-base):
Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
A. He was inspired by the Chinese writing system. B. He was inspired by the Japanese writing system. C. He was inspired by the Mongolian writing system. D. He was inspired by the Korean writing system.
- response: King Sejong the Great, the fourth king of the Joseon Dynasty in Korea, created Hangul in the 15th century with the aim of promoting literacy among the common people. Prior to Hangul, the Korean writing system was based on Classical Chinese characters, which were complex and difficult for the general population to learn.
King Sejong and his scholars were inspired by the need to create a writing system that was easy to learn and use, and that could accurately represent the sounds of the Korean language. They were not directly inspired by the Chinese, Japanese, or Mongolian writing systems, but rather by the desire to create a system that was uniquely Korean and accessible to all
Original model(microsoft/Phi-3-mini-4k-instruct):
Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?<|end|>
Other model(gpt-3.5-turbo-instruct - completions model):
Korean is written with an alphabet called Hangul, which was created in the 15th century by King Sejong the Great and his scholars. How King Sejong the Great created Hangul?
King Sejong the Great wanted to create a writing system that was easy to learn and use for the common people, as the existing writing system at the time, Chinese characters, were difficult to learn and only accessible to the elite. He believed that a writing system that was easy to learn would improve literacy rates and promote education among the common people.
To create Hangul, King Sejong the Great gathered a group of scholars and linguists and tasked them with creating a new writing system. They studied the sounds of the Korean language and based the alphabet on the shape and position of the mouth when making these sounds.
The alphabet was initially called Hunminjeongeum, which means "the correct sounds for the instruction of the people." It consisted of 28 letters, 17 consonants and 11 vowels, and was designed to be simple and easy to learn. The letters were also organized into syllable blocks, making it easier to read and write.
Hangul was officially promulgated in 1446 and was met with resistance from the elite who saw it as a threat to their status. However, King Sejong the Great continued to promote and encourage the use of Hangul, and it eventually became the primary writing system in Korea.
Today, Hangul is
Author's Note: It's still seems to be instruct-following model, but the response is more better than the original model as it doesn't have the <|end|> token.
Sample 2
Prompt:
South Korea has experienced many wars in modern times. For instance,
This model(devngho/phi-3-mini-4k-base):
South Korea has experienced many wars in modern times. For instance, the Korean War (1950-1953) was a conflict between North Korea and South Korea, with the United States and China supporting the former and the latter, respectively. The war ended in a stalemate, with the two Koreas remaining divided along the 38th parallel.
The Korean War had a significant impact on the country's economy. The war caused widespread destruction, with many cities and towns being destroyed. The war also disrupted trade and caused a shortage of resources. The South Korean government had to rely on foreign aid to rebuild the country.
One of the most significant impacts of the Korean War on the economy was the loss of human capital. Many young men were killed or injured during the war, and their families had to bear the burden of their loss. The war also caused a brain drain, with many skilled workers and professionals leaving the country to seek better opportunities abroad.
The Korean War also had a significant impact on the country's agriculture. The war disrupted the supply chain, and many farmers were unable to sell their produce. The government had to import
Original model(microsoft/Phi-3-mini-4k-instruct):
South Korea has experienced many wars in modern times. For instance, the Korean War (1950-1953) was a significant conflict that involved North and South Korea, as well as international forces. The war ended with an armistice agreement, but no formal peace treaty was signed, leaving the two Koreas technically still at war.
In addition to the Korean War, South Korea has faced other conflicts and tensions, such as the Korean DMZ Conflict (1966-1969) and the Jeju Uprising (1948). These events have shaped South Korea's history and continue to influence its relationships with neighboring countries.
Despite these conflicts, South Korea has made significant progress in terms of economic development, technological advancements, and cultural influence. The country has become a major player in the global economy, with a strong focus on industries such as electronics, automotive, and entertainment.
South Korea's success can be attributed to various factors, including its emphasis on education, innovation, and a strong work ethic. The country has also invested heavily in infrastructure and technology, which has
Other model(gpt-3.5-turbo-instruct - completions model):
South Korea has experienced many wars in modern times. For instance, the country was involved in the Korean War from 1950 to 1953, which was a conflict between North and South Korea. This war resulted in millions of casualties and the division of the Korean peninsula into two separate countries.
In addition, South Korea was also involved in the Vietnam War from 1964 to 1973, where it sent over 300,000 troops to support the United States and South Vietnam against North Vietnam and the Viet Cong. This war also resulted in thousands of casualties for South Korea.
In more recent times, South Korea has also faced tensions and conflicts with North Korea, including the Korean DMZ Conflict in 1966, the Korean Axe Murder Incident in 1976, and the Yeonpyeong Island shelling in 2010. These conflicts have resulted in casualties and heightened tensions between the two countries.
South Korea has also been involved in international conflicts, such as the Gulf War in 1991 and the War in Afghanistan from 2001 to 2014, where it sent troops to support the United States and its allies.
Overall, South Korea has a long history of experiencing wars and conflicts, both domestically and internationally. However, the country has also made efforts to promote peace and reconciliation, such as through the Sunshine Policy
Author's Note: These responses are very similar.
Sample 3
Prompt:
Hangul is a writing system made by King Sejong in
This model(devngho/phi-3-mini-4k-base):
Hangul is a writing system made by King Sejong in 1443. It is the official writing system of Korea. Hangul is a phonetic alphabet, which means that the letters represent the sounds of the Korean language.
The Creation of Hangul
King Sejong wanted to create a writing system that was easy to learn and use for the common people. He believed that the existing writing system, which was based on Chinese characters, was too difficult for the average person to learn.
The Structure of Hangul
Hangul is made up of 14 consonants and 10 vowels. These letters are combined to form syllables, which are the building blocks of the Korean language. The consonants and vowels are arranged in a square-like structure, with each row and column representing a different sound.
The Importance of Hangul
Hangul is an important part of Korean culture and identity. It is used in all aspects of life, from education to business to entertainment. The Korean government has made efforts to promote the use of Hangul and preserve its cultural significance.
Original model(microsoft/Phi-3-mini-4k-instruct):
Hangul is a writing system made by King Sejong in the 15th century. <|assistant|>
Other model(gpt-3.5-turbo-instruct - completions model):
Hangul is a writing system made by King Sejong in 1443. It is the official writing system of Korea and is used to write the Korean language. Hangul is unique in that it is a phonetic alphabet, meaning that each letter represents a specific sound. This makes it relatively easy to learn and use compared to other writing systems.
Hangul is made up of 14 consonants and 10 vowels, which can be combined to form syllables. The letters are arranged in a block-like structure, with each block representing a syllable. The consonants are written in a vertical line on the left side, while the vowels are written in a horizontal line on the top or bottom of the consonant.
One of the main reasons King Sejong created Hangul was to increase literacy among the common people. Before Hangul, the Korean language was written using Chinese characters, which were difficult to learn and only accessible to the elite. Hangul made it easier for people to learn how to read and write, leading to a significant increase in literacy rates in Korea.
Hangul has also played a significant role in preserving the Korean language and culture. During the Japanese occupation of Korea in the early 20th century, the use of Hangul was banned, and the Korean language was suppressed. However, after Korea gained independence,
Author's Note: Unlike the other samples, the model's response seems to be similar to general articles or gpt-3.5-turbo-instruct, not chat-style, as I hoped.
Sample 4
Prompt:
TPU(Tensor Processing Unit) was created by
This model(devngho/phi-3-mini-4k-base):
TPU(Tensor Processing Unit) was created by Google to accelerate the training of deep neural networks. TPUs are designed to be highly efficient for the matrix operations that are common in deep learning. They are similar to GPUs but are optimized for TensorFlow, Google's machine learning framework.
Here's a simple example of how you can use TPUs in TensorFlow to train a neural network:
First, you need to install the TensorFlow package with TPU support:
pip install tensorflow-gpu
Then, you can set up your TPU environment:
import tensorflow as tf
# Check if TPU is available
tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
print('Running on TPU ', tpu.master())
# Initialize the TPU
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
# Create a strategy for TPU distribution
strategy = tf.distribute.experimental.TPUStrategy(
Original model(microsoft/Phi-3-mini-4k-instruct):
TPU(Tensor Processing Unit) was created by Google.
TPU is a Google-designed accelerator that is optimized for training and inference of machine learning models.
I'm wondering if there is any open source implementation of TPU?
I've found this one: https://github.com/tensorflow/models/tree/master/official/transformer
But it seems to be a very small part of TPU.
- [Response]: Tensor Processing Units (TPUs) are custom-designed hardware accelerators developed by Google to efficiently run machine learning workloads, particularly those involving neural networks. While TPUs are proprietary and not open-source hardware, there are open-source software implementations that can run on TPUs or similar hardware architectures.
The TensorFlow ecosystem, which is Google's open-source machine learning framework, has made significant strides in providing support for TPUs. TensorFlow provides a TPU-optimized version of its runtime, which can be used to run TensorFlow models on TPUs.
Here are some open-source implementations and resources related to TPU
Other model(gpt-3.5-turbo-instruct - completions model):
TPU(Tensor Processing Unit) was created by Google for the purpose of accelerating machine learning workloads. It is a specialized chip designed specifically for deep learning tasks, such as training and inference of neural networks.
TPUs are highly efficient and can provide significant speedups compared to traditional CPUs and GPUs. They are optimized for matrix multiplication, which is a key operation in neural network computations. This allows them to process large amounts of data in parallel, making them ideal for training and running complex models.
One of the main advantages of TPUs is their scalability. They can be used individually or in clusters, allowing for even faster processing of large datasets. They also have a high memory bandwidth, which enables them to handle large models and datasets without running into memory constraints.
Another key feature of TPUs is their low power consumption. They are designed to be energy-efficient, making them a more environmentally friendly option compared to traditional hardware.
TPUs are primarily used in Google's cloud computing platform, Google Cloud, but they can also be accessed through Google Colab, a free online platform for running machine learning experiments. This makes TPUs accessible to a wider range of users, including researchers and developers who may not have access to specialized hardware.
Overall, TPUs have revolutionized the field of deep learning by providing a powerful and efficient hardware solution
Author's Note: The model provides a simple example of how to use TPUs in TensorFlow, which is a good response. Original model is Q&A style.
Sample 5
Prompt:
Hello!
This model(devngho/phi-3-mini-4k-base):
Hello! I'm your friendly AI chatbot, here to help you with your homework. Could you please tell me which subject and topic you need assistance with?
Conversation
User: Hey there! I'm working on a project for my history class, but it's a bit of a mix. We're looking at the impact of technology on society, but it's got to tie in with a historical event. Any ideas?
Chatbot: That sounds like an interesting project! How about exploring the impact of the printing press on the Reformation? The printing press was a significant technological advancement that greatly influenced the spread of ideas during the Reformation.
User: That's a good start, but I was thinking more along the lines of something more recent, like the internet.
Chatbot: Understood. You could examine the role of the internet in the Arab Spring. The widespread use of social media and the internet played a crucial role in organizing protests and disseminating information during that period.
User: Yeah, that's more like it. But I'
Original model(microsoft/Phi-3-mini-4k-instruct):
Hello! I'd be happy to help you with your SQL queries. Could you please describe your data or provide the table structure you're working with?
<|assistant|>
Other model(gpt-3.5-turbo-instruct - completions model):
Hello! I am a 22 year old female who is looking for a room to rent in the city of Toronto. I am a recent university graduate and will be starting a full-time job in the downtown area in September. I am a clean, responsible, and friendly individual who enjoys cooking, reading, and exploring the city. I am looking for a room in a shared house or apartment with other young professionals or students. My budget is around $800-1000 per month. Please contact me if you have a room available. Thank you!
Author's Note: The model's response is still instruct-following, it just doesn't use the <|assistant|> token. In contrast, gpt-3.5-turbo-instruct's response looks a normal introduction.
Bias, Risks, and Limitations
Since the base model is a instruct-following model, sometimes the model may generate chat-like responses (See Samples).
Training Details
Training Data
fineweb-edu (about 150,000 samples)
Training Procedure
- batch_size: 64
- lr: 3e-5
- lr_scheduler: cosine
- torch_dtype: bfloat16
- warmup_ratio: 0.2
- optimizer: adamw
- seed: 42
- gradient_accumulation: 1
- gradient_checkpointing: true
- FSDPv2 (FSDP via SPMD)
Compute Infrastructure
Google Cloud TPU
Hardware
TPU v4-32, took ~12 hours.
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). ⚡
Software
transformers~=4.41.2 torch~=2.3.0 torch_xla[tpu]~=2.3.0
Train Results
- train/loss: 2.22385830132309