<jupyter_start><jupyter_text>Practical Deep Learning for Coders Course - Lesson 3> "This blog-post series captures my weekly notes while I attend the [fastaiv5 course conducted by University of Queensland with fast.ai](https://itee.uq.edu.au/event/2022/practical-deep-learning-coders-uq-fastai). So off to week3 where we learn more how neural networks and say the C word while learning neural networks ..."- toc: true- branch: master- badges: true- hide_binder_badge: true- hide_deepnote_badge: true- comments: true- author: <NAME>- categories: [fastai, fastaicourse]                                                         - hide: false- search_exclude: false  Lesson SetupThere was a minor delay in streaming the lessons today, as today the sessions where being conducted in-person by Jeremy at University of Queensland. There were 130 people watching live inyoutube, while there was some notable absentees like [<NAME>](twitter.com/bhutanisanyam1/).Jeremy started the lesson by saying that usually lesson 1 and 2 are easy for everyone, while it'susually from lesson 3 things start getting hard. There is also a lesson 0 on how to do fast.ai?> youtube: https://youtu.be/gGxe2mN3kAgI had the previously written about [fastai lesson 0](https://kurianbenoy.com/2021-06-16-fastgroup-1/),where Jeremy mentioned about **How to do fast.ai lesson** through the following five steps:1. Watching lecture/book (watching the video first without trying anything)2. Running notebook and Experimentation (going through lesson notebooks and experimenting stuff)3. Reproduce Results (try with fastai clean notebook version, see if you are able to understand and do things on your own)4. Working on a different dataset (play with a different dataset, paraticipate in kaggle ...)Always studying done with other people is the best activity. So it's great to participate in studygroups like [Delft-fastai sessions](https://www.meetup.com/delft-fast-ai-study-group/).This week, Jeremy showcased various work based on which all got highest number of votes in share-my group section. My work also got featured 🙂.![image](https://user-images.githubusercontent.com/24592806/168640845-07859116-bf6b-48c7-af61-5ff8a65dc2fb.png)  Dogs vs Cat notebooks- which image models are the best?Today Jeremy featured, paperspace gradient platform. He has been using it for paperspace, and it's totally amazing. He got something done by them to update fastbookregularly.> Important: In lesson2 the main things, is not about taking a particular platform and deploying them through javascript websites or online applications. But the key thing is to undestand the concept. There are two pieces:> Note: 1. The Training piece by end of which you get a model.pkl file. Once you got that (train.ipynb)> Note: 2. Then part which takes inputs, spits out output ... this separate step is deploying (app.ipynb)Finding good image models, by [baselines results along with inference time](https://www.kaggle.com/code/jhoward/which-image-models-are-best/) will help us choose good architecture. He tried levit_models, which didn't work really great.From [13:52] he experiments with convnext tiny models from timm library. It got really good accuracy with almost 0.05 loss. At the moment for computer vision there are lot of good architectures, which beats resnets really well. In this case for predicting 37 breeds of dogs we can find categories in dataset using vocab of dataloaders in model.```labels = model.dls.vocab``` It's very important to understand what's in a model? Using `get_submodule` in pytorch we can look at the various neural networks, what is their input and ouput and each layers. [21:24] Let's explore the architecture of a [translation model which translates from their target form to english](https://huggingface.co/Helsinki-NLP/opus-mt-mul-en/tree/main).<jupyter_code>#hide-output
from transformers import AutoModelForSeq2SeqLM


model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-mul-en")<jupyter_output><empty_output><jupyter_text>**Looking at model architecture**<jupyter_code>#collapse_output
model<jupyter_output><empty_output><jupyter_text>**Looking at layer 1 `self_attn_layer_norm`**<jupyter_code>#collapse_output
attention_layer = model.get_submodule('model.encoder.layers.0.self_attn_layer_norm')
list(attention_layer.parameters())<jupyter_output><empty_output><jupyter_text>**Looking at shape of last layer**<jupyter_code>#collapse_output
final = model.get_submodule('model.decoder.layers.5.final_layer_norm')
final_paramaeters = list(final.parameters())
print(f"{final_paramaeters = }")<jupyter_output>final_paramaeters = [Parameter containing:
tensor([ 9.2454,  9.3895,  9.3544,  9.0685,  9.2224,  9.8569,  9.3900,  9.4416,
         9.4985,  9.2981,  9.5326,  9.2260,  8.8878,  9.4862,  9.5422,  9.3088,
         9.6653,  8.9836,  9.5670,  9.0307,  9.4179,  9.8929,  9.3411,  8.9442,
         8.3855,  9.0165,  9.5142,  9.5201,  9.2902,  9.5196,  8.8687,  9.3270,
         8.7709,  9.5791,  9.4227,  8.9457,  9.4278,  9.2320,  9.5537,  9.3045,
         9.2281,  9.1897,  8.9683,  9.3930,  9.1265,  9.2261,  9.1755,  9.2192,
         9.1531,  9.2323,  9.1581,  9.3413,  8.4585,  9.3836,  9.7359,  8.8970,
         9.4054,  8.9220,  9.2355,  9.6045,  9.6126,  9.4839,  9.2955,  9.2803,
         9.5649,  8.8892,  9.4749,  8.8119,  9.3922,  9.0771,  9.7973,  8.9035,
         9.7339,  9.1203,  9.5283,  8.9696,  8.4717,  9.3626,  9.3828,  7.9538,
         8.8453,  9.0190,  9.3108,  8.3297,  8.7236,  8.8562,  9.1680,  8.8641,
         7.8828,  8.7943,  8.4220,  8.8387,  9.3143,  8.1786,  9.1979,  9.064[...]
{'fraction_non_alphanumeric': 0.10021398002853067, 'fraction_numerical': 0.1011055634807418, 'mean_word_length': 3.9375, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 23, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '183769', 'license': 'Apache-2.0', 'n_tokens_mistral': 2152, 'n_tokens_neox': 1856, 'n_words': 626, 'path': '_notebooks/2022-05-10-fastai-53.ipynb', 'repo_name': 'kurianbenoy/ml-blog', 'size': '5622'}