v0.2

README.md

@@ -1,12 +1,22 @@
 ---
-library_name: transformers
 license: apache-2.0
+base_model: HuggingFaceTB/SmolLM-360M
+tags:
+- alignment-handbook
+- trl
+- sft
+datasets:
+- Magpie-Align/Magpie-Pro-300K-Filtered
+- bigcode/self-oss-instruct-sc2-exec-filter-50k
+- teknium/OpenHermes-2.5
+- HuggingFaceTB/everyday-conversations-llama3.1-2k
+library_name: transformers
 language:
 - en
 ---
 
 
-# SmolLM-Instruct
+# SmolLM-360M-Instruct
 
 <center>
     <img src="https://huggingface.co/datasets/HuggingFaceTB/images/resolve/main/banner_smol.png" alt="SmolLM" width="1100" height="600">
@@ -14,15 +24,29 @@ language:
 
 
 ## Model Summary
+Chat with the model at: https://huggingface.co/spaces/HuggingFaceTB/instant-smol
+
+SmolLM is a series of language models available in three sizes: 135M, 360M, and 1.7B parameters. 
+
+These models are trained on [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), a curated collection of high-quality educational and synthetic data designed for training LLMs. For further details, we refer to our [blogpost](https://huggingface.co/blog/smollm).
+
+To build SmolLM-Instruct, we finetune the base models on publicly available datasets.
+
+## Changelog
 
-SmolLM is a series of state-of-the-art small language models available in three sizes: 135M, 360M, and 1.7B parameters. These models are built on Cosmo-Corpus, a meticulously curated high-quality training dataset. Cosmo-Corpus includes Cosmopedia v2 (28B tokens of synthetic textbooks and stories generated by Mixtral), Python-Edu (4B tokens of educational Python samples from The Stack), and FineWeb-Edu (220B tokens of deduplicated educational web samples from FineWeb). For duther details, we refer to our blogpost TODO.
+|Release|Description|
+|-|-|
+|v0.1| Initial release of SmolLM-Instruct. We finetune on the permissive subset of the [WebInstructSub](https://huggingface.co/datasets/TIGER-Lab/WebInstructSub) dataset, combined with [StarCoder2-Self-OSS-Instruct](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k). Then, we perform DPO (Direct Preference Optimization) for one epoch on [HelpSteer](https://huggingface.co/datasets/nvidia/HelpSteer) for the 135M and 1.7B models, and [argilla/dpo-mix-7k](https://huggingface.co/datasets/argilla/dpo-mix-7k) for the 360M model.|
+|v0.2| We changed the finetuning mix to datasets more suitable for smol models. We train on a new dataset of 2k simple everyday conversations we generated by llama3.1-70B [everyday-conversations-llama3.1-2k](https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k/), [Magpie-Pro-300K-Filtere](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-300K-Filtered), [StarCoder2-Self-OSS-Instruct](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k), and a small subset of [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)|
 
-To build SmolLM-Instruct, we instruction tuned the models using publicly available permissive instruction datasets. We trained all three models for one epoch on the permissive subset of the WebInstructSub dataset, combined with StarCoder2-Self-OSS-Instruct. Following this, we performed DPO (Direct Preference Optimization) for one epoch: using HelpSteer for the 135M and 1.7B models, and argilla/dpo-mix-7k for the 360M model. We followed the training parameters from the Zephyr-Gemma recipe in the alignment handbook, but adjusted the SFT (Supervised Fine-Tuning) learning rate to 3e-4.
-[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+## Usage
 
-This is the SmolLM-360M-Instruct.
+### Local Applications
+⚡ For local applications, you can find optimized implementations of the model in MLC, GGUF and Transformers.js formats, in addition to fast in-browser demos in this collection: https://huggingface.co/collections/HuggingFaceTB/local-smollms-66c0f3b2a15b4eed7fb198d0
 
-### Generation
+We noticed that 4bit quantization degrades the quality of the 135M and 360M, so we use `q016` for MLC and ONNX/Transformers.js checkpoints for the WebGPU demos. We also suggest using temperature 0.2 and top-p 0.9.
+
+### Transformers
 ```bash
 pip install transformers
 ```
@@ -37,17 +61,32 @@ tokenizer = AutoTokenizer.from_pretrained(checkpoint)
 # for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
 model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
 
-messages = [{"role": "user", "content": "List the steps to bake a chocolate cake from scratch."}]
+messages = [{"role": "user", "content": "What is the capital of France."}]
 input_text=tokenizer.apply_chat_template(messages, tokenize=False)
 print(input_text)
 inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
-outputs = model.generate(inputs, max_new_tokens=100, temperature=0.6, top_p=0.92, do_sample=True)
+outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
 print(tokenizer.decode(outputs[0]))
 ```
 
-# Limitations
+### Chat in TRL
+You can also use the TRL CLI to chat with the model from the terminal:
+```bash
+pip install trl
+trl chat --model_name_or_path HuggingFaceTB/SmolLM-360M-Instruct --device cpu
+```
+
+## Limitations
+
+Additionally, the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data, we invite users to leverage them as assistive tools rather than definitive sources of information. We find that they can handle general knowledge questions, creative writing and basic Python programming. But they are English only and may have difficulty with arithmetics, editing tasks and complex reasoning. For more details about the models' capabilities, please refer to our [blog post](https://huggingface.co/blog/smollm).
 
-While SmolLM models have been trained on a diverse dataset including educational content and synthetic texts, they have limitations. The models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content. For a more comprehensive discussion of the models' capabilities and limitations, please refer to our full blog post.
+## Training parameters
+We train the models using the [alignement-handbook](https://github.com/huggingface/alignment-handbook) with the datasets mentioned in the changelog, using these parameters for v0.2:
+- 1 epoch
+- lr 1e-3
+- cosine schedule
+- warmup ratio 0.1
+- global batch size 262k tokens
 
 # Citation
 ```bash

all_results.json

@@ -1,22 +0,0 @@
-{
-    "epoch": 0.985781990521327,
-    "eval_logits/chosen": -0.5211005210876465,
-    "eval_logits/rejected": -0.7105662226676941,
-    "eval_logps/chosen": -442.4708557128906,
-    "eval_logps/rejected": -418.5675354003906,
-    "eval_loss": 0.6833909749984741,
-    "eval_rewards/accuracies": 0.59375,
-    "eval_rewards/chosen": -0.008601006120443344,
-    "eval_rewards/margins": 0.021825773641467094,
-    "eval_rewards/rejected": -0.03042677976191044,
-    "eval_runtime": 8.5409,
-    "eval_samples": 750,
-    "eval_samples_per_second": 87.812,
-    "eval_steps_per_second": 2.81,
-    "total_flos": 0.0,
-    "train_loss": 0.6860739244864538,
-    "train_runtime": 316.322,
-    "train_samples": 6750,
-    "train_samples_per_second": 21.339,
-    "train_steps_per_second": 0.164
-}

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "HuggingFaceTB/cosmo2-350M-webinst-sc2",
+  "_name_or_path": "HuggingFaceTB/SmolLM-360M",
   "architectures": [
     "LlamaForCausalLM"
   ],

eval_results.json

@@ -1,16 +0,0 @@
-{
-    "epoch": 0.985781990521327,
-    "eval_logits/chosen": -0.5211005210876465,
-    "eval_logits/rejected": -0.7105662226676941,
-    "eval_logps/chosen": -442.4708557128906,
-    "eval_logps/rejected": -418.5675354003906,
-    "eval_loss": 0.6833909749984741,
-    "eval_rewards/accuracies": 0.59375,
-    "eval_rewards/chosen": -0.008601006120443344,
-    "eval_rewards/margins": 0.021825773641467094,
-    "eval_rewards/rejected": -0.03042677976191044,
-    "eval_runtime": 8.5409,
-    "eval_samples": 750,
-    "eval_samples_per_second": 87.812,
-    "eval_steps_per_second": 2.81
-}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0c109192ffddcfc2be1ba4ae52a01e5713a98de94e98d90167e218aeebf919cc
+oid sha256:7f841de9bf656ea5af3c8d89e5647ee0f39121b28adf4202d1eec02f69619c5e
 size 723674912

onnx/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2496148b109c3bb6a3a8df92fea43c5647016ba0a2db08f89af30501e2cadf8c
-size 1448301209
+oid sha256:454394e1f92c1479bf71926b2cc845a3e29040c0844ba0d97ce693a390bca40c
+size 1448299818

onnx/model_bnb4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:537b03aa64065ce45f7bd19b39184e55f13ebe31965354c72b99c159526f8ea9
-size 366992362
+oid sha256:359ed0877d941aa2adf227aed8c209017dc0a7de6416ef322df6492356ba50e9
+size 366990971

onnx/model_fp16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6f18682159811608af15fb542ebe03f3fa7e544cf4e8bc12bd6852c3aca4e6b7
-size 724701643
+oid sha256:5e617d2597f06572ed87667f62a8a29e18bd5763fcb2606f7ad956b3f42d1cc6
+size 724678342

onnx/model_int8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:85af34637f211b2905193c8c08819d1c094db0c7a2f5c43984bdc3b04ecdad39
-size 363270670
+oid sha256:c42d79415353fe2d1470c2b3e27ad1dd84595cd2d31c93773d93e2df92ab5bcb
+size 363269280

onnx/model_q4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f27edbac285b53db0f3479698f02f401e88be5e1c865eec1689f668582a3bfa6
-size 386651434
+oid sha256:d8001535cd703e7c9afe6f6cf8f48e41ec6e162eb677ac39632787ad380ac2d7
+size 386650043

onnx/model_q4f16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f0adc35f1c264f54f5e03f65df6fa47cfebb1ef41e0442158112964b86df5b4
-size 272536796
+oid sha256:e1a788453e1393e8642f43ca729b7f2301ba61cc1f8ac1f1904c809869fc1ffb
+size 272513495

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:85af34637f211b2905193c8c08819d1c094db0c7a2f5c43984bdc3b04ecdad39
-size 363270670
+oid sha256:c42d79415353fe2d1470c2b3e27ad1dd84595cd2d31c93773d93e2df92ab5bcb
+size 363269280

onnx/model_uint8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db327d1cee0c3d4927e8640be2228264a6b5c038d22d797afaf71ddb37c29945
-size 363270784
+oid sha256:8f3309c58be313a8dce00de2f9d009e52c273d814ddc230e0a83e7cee8075c63
+size 363269392

runs/Jul15_21-49-55_ip-26-0-167-245/events.out.tfevents.1721080291.ip-26-0-167-245.4053728.0

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5f5dce01aa2e28781812105dc503156ef6f8285cc8c15e77a4b2f17cbfb8eb53
-size 9608

runs/Jul15_21-49-55_ip-26-0-167-245/events.out.tfevents.1721080634.ip-26-0-167-245.4053728.1

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:958e3bee85da7cc72b429819cff501ffb58ca95731aaf61bdefa71b132d36753
-size 815

special_tokens_map.json

@@ -1,29 +1,23 @@
 {
   "additional_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>"
+    {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
   ],
-  "bos_token": {
-    "content": "<|im_start|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "<|im_end|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<|im_end|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|im_end|>",
+  "pad_token": "<|im_end|>",
   "unk_token": {
     "content": "<|endoftext|>",
     "lstrip": false,

test_prompts.py

@@ -0,0 +1,106 @@
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+BASE_PATH = "/fsx/loubna/projects/alignment-handbook/recipes/cosmo2/sft/data"
+TEMPERATURE = 0.2
+TOP_P = 0.9
+
+CHECKPOINT = "loubnabnl/smollm-350M-instruct-add-basics"
+
+print(f"💾 Loading the model and tokenizer: {CHECKPOINT}...")
+device = "cuda"
+tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)
+model_s = AutoModelForCausalLM.from_pretrained(CHECKPOINT).to(device)
+
+print("🧪 Testing single-turn conversations...")
+L = [
+    "Hi",
+    "Hello",
+    "Tell me a joke",
+    "Who are you?",
+    "What's your name?",
+    "How do I make pancakes?",
+    "Can you tell me what is gravity?",
+    "What is the capital of Morocco?",
+    "What's 2+2?",
+    "Hi, what is 2+1?",
+    "What's 3+5?",
+    "Write a poem about Helium",
+    "Hi, what are some popular dishes from Japan?",
+]
+
+
+for i in range(len(L)):
+    print(f"🔮 {L[i]}")
+    messages = [{"role": "user", "content": L[i]}]
+    input_text = tokenizer.apply_chat_template(messages, tokenize=False)
+    inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
+    outputs = model_s.generate(
+        inputs, max_new_tokens=200, top_p=TOP_P, do_sample=True, temperature=TEMPERATURE
+    )
+    with open(
+        f"{BASE_PATH}/{CHECKPOINT.split('/')[-1]}_temp_{TEMPERATURE}_topp{TOP_P}.txt",
+        "a",
+    ) as f:
+        f.write("=" * 50 + "\n")
+        f.write(tokenizer.decode(outputs[0]))
+        f.write("\n")
+
+
+print("🧪 Now testing multi-turn conversations...")
+# Multi-turn conversations
+messages_1 = [
+    {"role": "user", "content": "Hi"},
+    {"role": "assistant", "content": "Hello! How can I help you today?"},
+    {"role": "user", "content": "What's 2+2?"},
+]
+messages_2 = [
+    {"role": "user", "content": "Hi"},
+    {"role": "assistant", "content": "Hello! How can I help you today?"},
+    {"role": "user", "content": "What's 2+2?"},
+    {"role": "assistant", "content": "4"},
+    {"role": "user", "content": "Why?"},
+]
+messages_3 = [
+    {"role": "user", "content": "Who are you?"},
+    {"role": "assistant", "content": "I am an AI assistant. How can I help you today?"},
+    {"role": "user", "content": "What's your name?"},
+]
+messages_4 = [
+    {"role": "user", "content": "Tell me a joke"},
+    {"role": "assistant", "content": "Sure! Why did the tomato turn red?"},
+    {"role": "user", "content": "Why?"},
+]
+messages_5 = [
+    {"role": "user", "content": "Can you tell me what is gravity?"},
+    {
+        "role": "assistant",
+        "content": "Sure! Gravity is a force that attracts objects toward each other. It is what keeps us on the ground and what makes things fall.",
+    },
+    {"role": "user", "content": "Who discovered it?"},
+]
+messages_6 = [
+    {"role": "user", "content": "How do I make pancakes?"},
+    {
+        "role": "assistant",
+        "content": "Sure! Here is a simple recipe for pancakes: Ingredients: 1 cup flour, 1 cup milk, 1 egg, 1 tbsp sugar, 1 tsp baking powder, 1/2 tsp salt. Instructions: 1. Mix all the dry ingredients together in a bowl. 2. Add the milk and egg and mix until smooth. 3. Heat a non-stick pan over medium heat. 4. Pour 1/4 cup of batter onto the pan. 5. Cook until bubbles form on the surface, then flip and cook for another minute. 6. Serve with your favorite toppings.",
+    },
+    {"role": "user", "content": "What are some popular toppings?"},
+]
+
+L = [messages_1, messages_2, messages_3, messages_4, messages_5, messages_6]
+
+for i in range(len(L)):
+    input_text = tokenizer.apply_chat_template(L[i], tokenize=False)
+    inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
+    outputs = model_s.generate(
+        inputs, max_new_tokens=200, top_p=TOP_P, do_sample=True, temperature=TEMPERATURE
+    )
+    with open(
+        f"{BASE_PATH}/{CHECKPOINT.split('/')[-1]}_temp_{TEMPERATURE}_topp{TOP_P}_MT.txt",
+        "a",
+    ) as f:
+        f.write("=" * 50 + "\n")
+        f.write(tokenizer.decode(outputs[0]))
+        f.write("\n")
+
+print("🔥 Done!")

train_results.json

@@ -1,9 +0,0 @@
-{
-    "epoch": 0.985781990521327,
-    "total_flos": 0.0,
-    "train_loss": 0.6860739244864538,
-    "train_runtime": 316.322,
-    "train_samples": 6750,
-    "train_samples_per_second": 21.339,
-    "train_steps_per_second": 0.164
-}

trainer_state.json

@@ -1,132 +0,0 @@
-{
-  "best_metric": null,
-  "best_model_checkpoint": null,
-  "epoch": 0.985781990521327,
-  "eval_steps": 100,
-  "global_step": 52,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "epoch": 0.018957345971563982,
-      "grad_norm": 3.021558692298453,
-      "learning_rate": 8.333333333333333e-07,
-      "logits/chosen": -0.5616407990455627,
-      "logits/rejected": -0.42516714334487915,
-      "logps/chosen": -373.83282470703125,
-      "logps/rejected": -493.9554748535156,
-      "loss": 0.6931,
-      "rewards/accuracies": 0.0,
-      "rewards/chosen": 0.0,
-      "rewards/margins": 0.0,
-      "rewards/rejected": 0.0,
-      "step": 1
-    },
-    {
-      "epoch": 0.1895734597156398,
-      "grad_norm": 3.23447571449701,
-      "learning_rate": 4.907293218369499e-06,
-      "logits/chosen": -0.1685400903224945,
-      "logits/rejected": -0.19170813262462616,
-      "logps/chosen": -464.29693603515625,
-      "logps/rejected": -501.8587341308594,
-      "loss": 0.6936,
-      "rewards/accuracies": 0.4097222089767456,
-      "rewards/chosen": 0.0008683237829245627,
-      "rewards/margins": 0.0007554867188446224,
-      "rewards/rejected": 0.00011283738422207534,
-      "step": 10
-    },
-    {
-      "epoch": 0.3791469194312796,
-      "grad_norm": 3.9389946840001264,
-      "learning_rate": 3.941700805287169e-06,
-      "logits/chosen": -0.27595698833465576,
-      "logits/rejected": -0.21783480048179626,
-      "logps/chosen": -438.6907653808594,
-      "logps/rejected": -475.0658264160156,
-      "loss": 0.6894,
-      "rewards/accuracies": 0.4937500059604645,
-      "rewards/chosen": -0.003008360043168068,
-      "rewards/margins": 0.0011918289819732308,
-      "rewards/rejected": -0.00420018844306469,
-      "step": 20
-    },
-    {
-      "epoch": 0.5687203791469194,
-      "grad_norm": 3.2779169381175386,
-      "learning_rate": 2.3293939665883233e-06,
-      "logits/chosen": -0.012437907047569752,
-      "logits/rejected": -0.20707044005393982,
-      "logps/chosen": -476.6265563964844,
-      "logps/rejected": -474.9481506347656,
-      "loss": 0.6873,
-      "rewards/accuracies": 0.53125,
-      "rewards/chosen": -0.01446505170315504,
-      "rewards/margins": 0.007696834392845631,
-      "rewards/rejected": -0.02216188609600067,
-      "step": 30
-    },
-    {
-      "epoch": 0.7582938388625592,
-      "grad_norm": 2.8592366061141954,
-      "learning_rate": 7.936171419533653e-07,
-      "logits/chosen": -0.4586310386657715,
-      "logits/rejected": -0.5929983854293823,
-      "logps/chosen": -422.44293212890625,
-      "logps/rejected": -402.6429748535156,
-      "loss": 0.6827,
-      "rewards/accuracies": 0.574999988079071,
-      "rewards/chosen": -0.018473980948328972,
-      "rewards/margins": 0.014362658374011517,
-      "rewards/rejected": -0.03283664211630821,
-      "step": 40
-    },
-    {
-      "epoch": 0.9478672985781991,
-      "grad_norm": 3.296382116125289,
-      "learning_rate": 2.3285134909173113e-08,
-      "logits/chosen": 0.08067023754119873,
-      "logits/rejected": 0.19673587381839752,
-      "logps/chosen": -483.49212646484375,
-      "logps/rejected": -536.2080688476562,
-      "loss": 0.679,
-      "rewards/accuracies": 0.5874999761581421,
-      "rewards/chosen": -0.017066333442926407,
-      "rewards/margins": 0.01992700807750225,
-      "rewards/rejected": -0.03699334338307381,
-      "step": 50
-    },
-    {
-      "epoch": 0.985781990521327,
-      "step": 52,
-      "total_flos": 0.0,
-      "train_loss": 0.6860739244864538,
-      "train_runtime": 316.322,
-      "train_samples_per_second": 21.339,
-      "train_steps_per_second": 0.164
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 52,
-  "num_input_tokens_seen": 0,
-  "num_train_epochs": 1,
-  "save_steps": 500,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": false,
-        "should_training_stop": false
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 0.0,
-  "train_batch_size": 2,
-  "trial_name": null,
-  "trial_params": null
-}

training_args.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:753772c371abf783b3f96c5ddc2c75471f7281c7efc51b1f2c0c1b6ddd4988f7
-size 6520