<jupyter_start><jupyter_text>![license_header_logo](../../../images/license_header_logo.png)> **Copyright (c) 2021 <EMAIL>ifAI Sdn. Bhd.**This program is part of OSRFramework. You can redistribute it and/or modifyit under the terms of the GNU Affero General Public License as published bythe Free Software Foundation, either version 3 of the License, or(at your option) any later version.This program is distributed in the hope that it will be usefulbut WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See theGNU Affero General Public License for more details.You should have received a copy of the GNU Affero General Public Licensealong with this program.  If not, see .  IntroductionThis handson is to demonstrate how to implement the tools, transformers and models from the HuggingFace library to create our own Machine Translation Model.Link to the HuggingFace Library: https://huggingface.co/models?sort=downloadsYou may check out slides Day 12 - Pretrained Model for NLP & Generalized Language Model, to get a detailed explanation and walkthrough on this handson.The walkthrough includes:1. Explaination of code2. How to find the model we want from HuggingFace and implement them into the code  What we will accomplish?1. Use transformers in HuggingFace2. Import the correct model from HuggingFace3. Create your own Machine Translation model  InstructionsRead the code and execute them according to the instructions provided. If you are having trouble understanding the code, you may take a look at slides, Day 12 - Pretrained Model for NLP & Generalized Language Model, Machine Translation Handson to get a better understanding.  Part 1: Code and its explanationFirst, we will install the required libraries.torch refers to PyTorch library and transformers refers to the HuggingFace transformers library.We need to install PyTorch in order to utilize the HuggingFace models and transformers.If you already have installed, you can skip this step. AutoModelForSeq2SeqLM is where machine translation models fall under in the HuggingFace library.AutoTokenizer is where we can define tokenizers from the HuggingFace library.Pipeline is a method where we can automate the workflow to produce machine learning model.<jupyter_code>from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline<jupyter_output><empty_output><jupyter_text>model is where we will be defining the machine translation modelwe are importing a pretrained model (indicated by from_pretraied) that is the Helsinki modeltokenizer is where we define tokenizer, also this is a tokenizer from pretrained Helsinki modeltranslation is where we call the pipeline method to automate the machine translation workflowhere we defined what process it is going to in the parametersfirst parameter, “translation_mul_to_en” means translation of multi language to englishsecond parameter, model=model is just us inserting the model we already initialized abovethird parameter, tokenizer=tokenizer is also just us inserting the tokenizer already initialized above<jupyter_code>model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-mul-en")
tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-mul-en")

translation = pipeline("translation", model=model, tokenizer=tokenizer)<jupyter_output><empty_output><jupyter_text>text is obviously the text that we want to translatetranslated_text is where we define translated text (Malay to English translated text)the translation method will translate the text to Englishthis method will return a dict, so we want to print only the content of the first element[0] in the dict tagged with ‘translation_text’<jupyter_code>text = "Nama saya Micheal, siapakah nama awak?"
translated_text = translation(text)[0]['translation_text']

print(translated_text)<jupyter_output>My name is Michael. What's your name?<jupyter_text>Part 2: Choosing the proper model from HuggingFaceIn this task, you are required to go to the hugging face website to look for the model that can translate english to chinese.HuggingFace website: https://huggingface.co/models?sort=downloadsIf you are having trouble with this task, take a look at slides Day 12 - Pretrained Model for NLP & Generalized Language Model, to get a detailed explanation and walkthrough on this handson.You will have to find the appropriate model, copy the name of the model provided and paste it into the model and tokenizer parameters.<jupyter_code>from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline

# model = AutoModelForSeq2SeqLM.from_pretrained("COPY AND PASTE THE MODEL NAME HERE")
# tokenizer = AutoTokenizer.from_pretrained("COPY AND PASTE THE MODEL NAME HERE")

# translation = pipeline("translation", model=model, tokenizer=tokenizer)

# text = "Hi, how are you?"
# translated_text = translation(text)[0]['translation_text']

# print(translated_text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04887901434053726, 'fraction_numerical': 0.005251464350636235, 'mean_word_length': 6.0042432814710045, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '145846', 'license': 'Apache-2.0', 'n_tokens_mistral': 1265, 'n_tokens_neox': 1211, 'n_words': 607, 'path': 'nlp-labs/Day_12/Machine Translation with HuggingFace/Machine Translation Handson.ipynb', 'repo_name': 'synthethic04/NLP-Traininglabs', 'size': '4948'}