BERT HuggingFace Model Deployment using Kubernetes [ Github Repo]  03/07/2024

Github Repo : https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/MLops/Model_Deployment/Bert_Kubernetes_deployment   Model development is useless if you dont deploy it to production  which comes with a lot of issues of scalability and portability.   I have deployed a basic BERT model from the huggingface transformer on Kubernetes with the help of docker  which will give a feel of how to deploy and manage pods on production.   Model Serving and Deployment:ML Pipeline:Workflow:   Model server (using FastAPI  uvicorn) for BERT uncased model    Containerize model and inference scripts to create a docker image    Kubernetes deployment for these model servers (for scalability)  Testing   Components:Model serverUsed BERT uncased model from hugging face for prediction of next word [MASK]. Inference is done using transformer-cli which uses fastapi and uvicorn to serve the model endpoints   Server streaming:   Testing: (fastapi docs)   http://localhost:8888/docs/   { output: [ { score: 0.21721847355365753  token: 2204  token_str: good  sequence: today is a good day }  { score: 0.16623663902282715  token: 2047  token_str: new  sequence: today is a new day }  { score: 0.07342924177646637  token: 2307  token_str: great  sequence: today is a great day }  { score: 0.0656224861741066  token: 2502  token_str: big  sequence: today is a big day }  { score: 0.03518620505928993  token: 3376  token_str: beautiful  sequence: today is a beautiful day } ]   ContainerizationCreated a docker image from huggingface GPU base image and pushed to dockerhub after testing.   Testing on docker container:   You can directly pull the image vaibhaw06/bert-kubernetes:latest   K8s deploymentUsed minikube and kubectl commands to create a single pod container for serving the model by configuring deployment and service config   deployment.yaml   apiVersion: apps/v1 kind: Deployment metadata: name: bert-deployment labels: app: bertapp spec: replicas: 1 selector: matchLabels: app: bertapp template: metadata: labels: app: bertapp spec: containers: - name: bertapp image: vaibhaw06/bert-kubernetes ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: bert-service spec: type: NodePort selector: app: bertapp ports: - protocol: TCP port: 8080 targetPort: 8080 nodePort: 30100Setting up minikube and running pods using kubectl and deployment.yaml   minikube start kubectl apply -f deployment.yamlFinal Testing:kubectl get allIt took around 15 mins to pull and create container pods.   kubectl image listkubectl get svcminikube service bert-serviceAfter running the last command minikube service bert-service  you can verify the resulting deployment on the web endpoint.   Find the GitHub Link: https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/MLops/Model_Deployment/Bert_Kubernetes_deployment   If you have any questions  ping me on my LinkedIn: https://www.linkedin.com/in/vaibhaw-khemka-a92156176/   Follow ML Umbrella for more such detailed  actionable projects.   Future Extension:Scaling with pod replicas and load balancer -   Self-healing


Named Entity Recognition in Ecommerce Industry  Custom model [Github Repo]  03/07/24

Github Repo: https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/NLP/Product-Categorization   From e-commerce to Customer support  all businesses require some kind of NER model to process huge amounts of texts from users.   To automate this whole  one requires NER models to extract relevant and important entities from text.   Final Result/OutputInput text = EL D68 (Green  32 GB) 3 GB RAM [3 GB RAM U+007C 32 GB ROM U+007C Expandable Upto 128 GB  15.46 cm (6.088 inch) Display  13MP Rear Camera U+007C 8MP Front Camera  4000 mAh Battery  Quad-Core Processor]   Output =   Green ->>>> COLOR 32 GB ->>>> STORAGE 3 GB RAM ->>>> RAM 3 GB RAM ->>>> RAM 32 GB ROM ->>>> STORAGE Expandable Upto 128 GB ->>>> EXPANDABLE_STORAGE 15.46 cm (6.088 inch) ->>>> SCREEN_SIZE 13MP Rear Camera ->>>> BACK_CAMERA 8MP Front Camera ->>>> FRONT_CAMERA 4000 mAh Battery ->>>> BATTERY_CAPACITY Quad-Core Processor ->>>> PROCESSOR_CORE   Data PreparationA tool for creating this dataset (https://github.com/tecoholic/ner-annotator)    Snapshot for the dataset for Mobile phone product description on Amazon:   A single record of the Data:   Converting into proper Spacy span format:The proper format that Spacy Ner model understands   import jsonlines  json file_path = Training Data/Mobile/Mobile_training.jsonl laptop_classes = [RAM STORAGE BATTERY CAPACITY PROCESSOR_TYPE SCREEN_SIZE REFRESH_RATE SCREEN_TYPE BACK_CAMERA FRONT_CAMERA] with jsonlines.open(file_path) as reader: output_json = {classes: laptop_classes  annotations: []} # Iterate over each line (JSON object) for obj in reader: processed_obj = [obj[text] {entities:obj[label]}] output_json[annotations].append(processed_obj) # Save the output JSON to a new file with open('Training Data/Mobile/Mobile_annotations.json'  'w') as f: json.dump(output_json  f  indent=None)Above is the code for converting into proper data format. Check out jupyter notebook: NER_model_Mobile.ipynb   Final pandas dataframe from processed data:   Splitting the dataset  10% test### Split the data from sklearn.model_selection import train_test_split train  test = train_test_split(df  test_size=0.1) train.head()Create spacy DocBin objects from annotated data to train Spacy NER model:import spacy from spacy.tokens import DocBin from tqdm import tqdm # Define a function to create spaCy DocBin objects from the annotated data def get_spacy_doc(data): # Create a blank spaCy pipeline nlp = spacy.blank('en') db = DocBin() # Initialize a counter for None spans none_spans = 0 spans = 0 for index  row in data.iterrows(): # Get the text and annotations text = row[Description] annotations = row[Annotations] # Check if the text is not empty if not text: continue # Process the text and annotations doc = nlp(text) if doc is None: print(fFailed to process text: {text}) continue ents = [] for start  end  label in annotations: if start < 0 or end < 0: print(fInvalid annotation: {start}  {end}  {label}) continue #print(text) span = doc.char_span(start  end  label=label) if span is None: print(fFailed to create span for annotation: {start}  {end}  {label}) none_spans += 1 continue else: spans+=1 ents.append(span) doc.ents = ents #Add the processed document to the DocBin db.add(doc) print(fNumber of None spans: {none_spans}) print(fNumber of spans: {spans}) return dbModellingArchitecture:The basic architecture for all spacy models:   Reference: https://explosion.ai/blog/deep-learning-formula-nlp   [Embed]HashEmbed  Sub-word features than character based richer representation and arbitrary sized vocabulary  Can use Word2vec/Glove etc   [Encode]  Context-independent to context-dependent using LSTM or CNN.   [Attend]  Attention mechanism by Key  Value pair  and context vectors   [Predict]  MLP   Tok2vec model [example]:   https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py (Built using thinc framework)   NER Model  Transition-Based:   State(all three stack  buffer  and output) and Action   Structure Prediction.   The above shows how the transition-based approach works with stack  buffer  output  and Transition/action.   Reference: https://www.microsoft.com/en-us/research/video/transition-based-natural-language-processing/   The above shows How stacked LSTM works for encoding for all states and actions.   The final Prediction from MLP is the Multiclassification task with labels as SHIFT  OUT  and REDUCE   Spacy model layer and Config Mapping:   Example of a tok2vec config:   Model in thinc framework:   Respective config for the model:   Thinc deep learning framework is used as a backend to build spacy models instead of pytorch or TensorFlow.   Difference between normal pytorch and spacy models. => Spacy(easy  reliable and productionable)   The user can define and create this model using a configuration file for any task: NER  Tok2Vec  Tagger  Dependency Parser  Sentiment etc   One can also create thinc models and wrap around pytorch and TensorFlow. I will build it next blog.   NER Config file created here:   Reference: https://spacy.io/usage/training   config_ner.cfg :   [paths] train = null dev = null vectors = en_core_web_lg init_tok2vec = null [system] gpu_allocator = null seed = 0 [nlp] lang = en pipeline = [tok2vec ner] batch_size = 1000 disabled = [] before_creation = null after_creation = null after_pipeline_creation = null tokenizer = {@tokenizers:spacy.Tokenizer.v1} vectors = {@vectors:spacy.Vectors.v1} [components] [components.ner] factory = ner incorrect_spans_key = null moves = null scorer = {@scorers:spacy.ner_scorer.v1} update_with_oracle_cut_size = 100 [components.ner.model] @architectures = spacy.TransitionBasedParser.v2 state_type = ner extra_state_tokens = false hidden_width = 64 maxout_pieces = 2 use_upper = true nO = null [components.ner.model.tok2vec] @architectures = spacy.Tok2VecListener.v1 width = ${components.tok2vec.model.encode.width} upstream = * [components.tok2vec] factory = tok2vec [components.tok2vec.model] @architectures = spacy.Tok2Vec.v2 [components.tok2vec.model.embed] @architectures = spacy.MultiHashEmbed.v2 width = ${components.tok2vec.model.encode.width} attrs = [NORM PREFIX SUFFIX SHAPE] rows = [5000 1000 2500 2500] include_static_vectors = true [components.tok2vec.model.encode] @architectures = spacy.MaxoutWindowEncoder.v2 width = 256 depth = 8 window_size = 1 maxout_pieces = 3 [corpora] [corpora.dev] @readers = spacy.Corpus.v1 path = ${paths.dev} max_length = 0 gold_preproc = false limit = 0 augmenter = null [corpora.train] @readers = spacy.Corpus.v1 path = ${paths.train} max_length = 0 gold_preproc = false limit = 0 augmenter = null [training] dev_corpus = corpora.dev train_corpus = corpora.train seed = ${system.seed} gpu_allocator = ${system.gpu_allocator} dropout = 0.1 accumulate_gradient = 1 patience = 1600 max_epochs = 0 max_steps = 20000 eval_frequency = 200 frozen_components = [] annotating_components = [] before_to_disk = null before_update = null [training.batcher] @batchers = spacy.batch_by_words.v1 discard_oversize = false tolerance = 0.2 get_length = null [training.batcher.size] @schedules = compounding.v1 start = 100 stop = 1000 compound = 1.001 t = 0.0 [training.logger] @loggers = spacy.ConsoleLogger.v1 progress_bar = false [training.optimizer] @optimizers = Adam.v1 beta1 = 0.9 beta2 = 0.999 L2_is_weight_decay = true L2 = 0.01 grad_clip = 1.0 use_averages = false eps = 0.00000001 learn_rate = 0.001 [training.score_weights] ents_f = 1.0 ents_p = 0.0 ents_r = 0.0 ents_per_type = null [pretraining] [initialize] vectors = ${paths.vectors} init_tok2vec = ${paths.init_tok2vec} vocab_data = null lookups = null before_init = null after_init = null [initialize.components] [initialize.tokenizer]Output and Evaluation:Evaluation is done based on ENTS_P(Precision)  ENTS_R(Recall) and ENTS_F (F-Score).   After the 15th epoch Final ENTS_F is 57.64  which can be improved by providing more data for this case.   Intuition for Evaluation:We evaluate the NER model based on Span-Identification and Span-Prediction.   Span-Identification:   https://cees-roele.medium.com/custom-evaluation-of-spans-in-spacy-f1f2e7a99ad8   As discussed  NER is a multiclass Classification problem with SHIFT  OUT  and REDUCE as output. But we evaluate our models only based on REDUCE.   The above picture shows how Precision  Recall  and F-Score are calculated.   The code used for evaluating PRF (Precision-Recall-Fscore) by spacy:   def get_ner_prf(examples: Iterable[Example]  **kwargs) -> Dict[str  Any]: Compute micro-PRF and per-entity PRF scores for a sequence of examples. score_per_type = defaultdict(PRFScore) for eg in examples: if not eg.y.has_annotation(ENT_IOB): continue golds = {(e.label_  e.start  e.end) for e in eg.y.ents} align_x2y = eg.alignment.x2y for pred_ent in eg.x.ents: if pred_ent.label_ not in score_per_type: score_per_type[pred_ent.label_] = PRFScore() indices = align_x2y[pred_ent.start : pred_ent.end] if len(indices): g_span = eg.y[indices[0] : indices[-1] + 1] # Check we aren't missing annotation on this span. If so  # our prediction is neither right nor wrong  we just # ignore it. if all(token.ent_iob != 0 for token in g_span): key = (pred_ent.label_  indices[0]  indices[-1] + 1) if key in golds: score_per_type[pred_ent.label_].tp += 1 golds.remove(key) else: score_per_type[pred_ent.label_].fp += 1 for label  start  end in golds: score_per_type[label].fn += 1 totals = PRFScore() for prf in score_per_type.values(): totals += prf if len(totals) > 0: return { ents_p: totals.precision  ents_r: totals.recall  ents_f: totals.fscore  ents_per_type: {k: v.to_dict() for k  v in score_per_type.items()}  } else: return { ents_p: None  ents_r: None  ents_f: None  ents_per_type: None  }Reference: https://github.com/explosion/spaCy/blob/master/spacy/scorer.py#L760   Span Prediction :   There are 9 different entires like [RAM  STORAGE  BATTERY CAPACITY  PROCESSOR_TYPE  SCREEN_SIZE  REFRESH_RATE  SCREEN_TYPE  BACK_CAMERA  FRONT_CAMERA] to predict for REDUCE class.   It uses categorical crossentropy loss function to optimize NER models (More details in later blogs)   Testing and Final Results:Input text = EL D68 (Green  32 GB) 3 GB RAM [3 GB RAM U+007C 32 GB ROM U+007C Expandable Upto 128 GB  15.46 cm (6.088 inch) Display  13MP Rear Camera U+007C 8MP Front Camera  4000 mAh Battery  Quad-Core Processor]   Output =   Green ->>>> COLOR 32 GB ->>>> STORAGE 3 GB RAM ->>>> RAM 3 GB RAM ->>>> RAM 32 GB ROM ->>>> STORAGE Expandable Upto 128 GB ->>>> EXPANDABLE_STORAGE 15.46 cm (6.088 inch) ->>>> SCREEN_SIZE 13MP Rear Camera ->>>> BACK_CAMERA 8MP Front Camera ->>>> FRONT_CAMERA 4000 mAh Battery ->>>> BATTERY_CAPACITY Quad-Core Processor ->>>> PROCESSOR_CORE   Github Link: https://github.com/vaibhawkhemka/ML-Umbrella/tree/main/NLP/Product-Categorization   Thanks for reading the blog.   If you have any questions  hit me up on my LinkedIn: https://www.linkedin.com/in/vaibhaw-khemka-a92156176/   References for modeling:   https://explosion.ai/blog/deep-learning-formula-nlp => Embed  Encode  Attend and Predict => Position is imp in sequence in text.   https://support.prodi.gy/t/spacy-ner-models-architecture-details/4336   https://github.com/explosion/spaCy/blob/master/spacy/ml/models/tok2vec.py   https://spacy.io/usage/layers-architectures   https://spacy.io/api/architectures#CharacterEmbed   Understanding span:   https://spacy.io/api/span


AdaBoost Explained From Its Original Paper

This publication is meant to show a very popular ML algorithm in complete detail  how it works  the math behind it  how to execute it in Python and an explanation of the proofs of the original paper. There will be math and code  but it is written in a way that allows you to decide which are the fun parts.   A bit on the origins of the algorithm: It was proposed by Yoav Freund and Robert E. Schapire in a 1997 paper  A Decision-Theoretic Generalization of On-Line Learning and an Application to Boostinga beautiful and brilliant publication for an effective and useful algorithm.   Lets start with the pros  cons  and uses of AdaBoost.   Advantages: improves performance and achieves higher accuracy than a single model. It reduces overfitting compared to some other machine learning algorithms.   Disadvantages: AdaBoost can be sensitive to noisy data and outliers. It requires careful tuning  and the performance can depend on the choice of weak learners and the number of iterations. It cannot be parallelized (or only partially)  since each predictor can only be trained after the previous predictor has been trained and evaluated. As a result  it does not scale as well as bagging or pasting.   Applications: image recognition  text classification  fraud detection  predictive modeling.   Introduction  what is ensemble learning and boosting?Python script with an Ada Boost algorithm  lets go straight to using this toolAda Boost explanation  the math on how it worksAda Boost example  simplifying the math  an example of one iterationReferencesIntroductionLets talk a bit about the wisdom of the crowd. Wisdom of the crowd is a phenomenon that suggests that the collective judgment of a diverse number of people is often surprisingly accurate. This mainly occurs because of the central limit theorem  which states that when you take an average of a large number of independent observations  the distribution will center around the true value.   Lets explain this with an example. What if there was a competition where people had to guess how many bubble gum pieces were in a jar? Thousands of different (independent) people will guess; some might be close  and others will be quite far from the true number  but once we calculate the average of the guesses  we will be quite close to the actual number of bubble gum balls  this my friends is the wisdom of the crowd.   How does this apply to Machine Learning?   If we have many predictors (decision trees  other classifiers or regressors) and we aggregate the predictions of this group  they will often perform better than the best individual predictor. A group of predictors is called an ensemble  thus  this technique is called Ensemble Learning.   AdaBoost belongs to a method called boosting. Boosting refers to any ensemble method that combines several weak learners (simple models) into a strong learner (a more accurate model).   There are many boosting methods  the most popular by far are Ada Boosting and Gradient Boosting.   Ada Boost with Python and Scikit-LearnPart 1: data preparationWe create a dummy dataset and separate the data into train and test.   import numpy as np from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split # Generate a random dataset (for example purposes) X  y = make_classification(n_samples=100  n_features=2  n_informative=2  n_redundant=0  random_state=42) # Split the dataset into training and testing sets X_train  X_test  y_train  y_test = train_test_split(X  y  test_size=0.3  random_state=42)Part 2: AdaBoost with Decision Trees (1 branch)First  lets understand the possible parameters in Scikit-learns AdaBoostClassifier:   estimator: The base estimator from which the boosted ensemble is built. Usually  a decision tree with a max depth 1 (a weak learner).n_estimators: The maximum number of estimators at which boosting is terminated.learning rate: Weight applied to each classifier at each boosting iteration. A higher learning rate increases the contribution of each classifier.random_state: Controls the random seed given at each estimator at each boosting iteration.from sklearn.ensemble import AdaBoostClassifier from sklearn.tree import DecisionTreeClassifier # Create the AdaBoost classifier # Notice that the depth of the decision tree is 1 base_estimator = DecisionTreeClassifier(max_depth=1) ada_boost = AdaBoostClassifier(estimator=base_estimator  n_estimators=50  learning_rate=1.0  random_state=42) # Train the classifier ada_boost.fit(X_train  y_train) # Make predictions y_pred = ada_boost.predict(X_test)Part 3: Model evaluationWe measure the metrics of the model. Interpretation of these metrics will be seen in a different article.   from sklearn.metrics import accuracy_score  classification_report # Evaluate the classifier accuracy = accuracy_score(y_test  y_pred) print(f'Accuracy: {accuracy:.2f}') print('Classification Report:') print(classification_report(y_test  y_pred))Accuracy: 0.97 Classification Report: precision recall f1-score support 0 1.00 0.94 0.97 16 1 0.93 1.00 0.97 14 accuracy 0.97 30 macro avg 0.97 0.97 0.97 30 weighted avg 0.97 0.97 0.97 30Part 4: Plotting Resultsimport matplotlib.pyplot as plt # Plotting the decision boundary x_min  x_max = X[:  0].min() - 1  X[:  0].max() + 1 y_min  y_max = X[:  1].min() - 1  X[:  1].max() + 1 xx  yy = np.meshgrid(np.arange(x_min  x_max  0.01)  np.arange(y_min  y_max  0.01)) Z = ada_boost.predict(np.c_[xx.ravel()  yy.ravel()]) Z = Z.reshape(xx.shape) plt.contourf(xx  yy  Z  alpha=0.3) plt.scatter(X[:  0]  X[:  1]  c=y  edgecolors='k'  marker='o') plt.title('AdaBoost Decision Boundary') plt.xlabel('Feature 1') plt.ylabel('Feature 2') plt.show()Ada Boost ExplanationIn this section  we explain the key concepts and how an iteration works (a bit of math included  folks.   AdaBoost  short for Adaptive Boosting  is a machine learning algorithm that is used to improve the performance of other machine learning algorithms. We will define a few key concepts to explain how it works:   Weak Learners: models that perform slightly better than random guessing. Decision trees with one split are often used.Boosting: the process of combining multiple weak learners to form a strong learner. Each learner has a weight based on the performance of the previous learners.Weight Adjustment: First  all data points have equal weights. After each iteration  the weight of incorrectly classified points is increased; that way  the learner focuses more on the difficult cases.Combining Learners: the final model is a weighted sum of all the weak learners; each learners contribution to the final model is based on its accuracy  and more accurate learners are given higher weights.Algorithm stepsThe image shows how the algorithm improves on each iteration on separating between the blue and red dots. Lets find out how each step works.   Initialize Weights:Assign equal weights to all data points (each predictor).2. Train Weak Learner and Calculate Weighted Error   Train a weak learner on the weighted dataset (h_t).Calculate the error rate of the weak learner.3. Calculate Alpha  the learner's weight   4. Update weights   5. Combine weak learners   Example of one iterationInitialize weights   Train Weak Learner and Calculate Weighted Error   Calculate Alpha   Update weights   This process continues for each iteration.   ReferencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting (1997)  Yoav Freund and Robert E. SchapireHands-On Machine Learning with Scikit-Learn  Keras  and TensorFlow (2019)  Aurelin GernSklearn documentation


Bias in Natural Language Processing (NLP)

The rising popularity of natural language processing (NLP) and machine learning technologies underscores the importance of recognizing their role in shaping societal biases and stereotypes. While NLP applications have achieved success in modeling tasks like sentiment analysis  machine translation  and text summarization  these models can perpetuate societal biases present in their training datasets. These biases pose significant threats to equity  justice  and democracy.   In this article  we will discuss how NLP reflects and amplifies societal prejudices  explore their consequences  and outline steps that companies providing natural language services need to take toward mitigating them.   Bias is favoring one person  thing  or group over another unfairly. Bias is not programmed into natural language processing models; instead  it implicitly creeps into the model through the statistical patterns in language data it learns from. The training data may incorporate societal prejudices and derogatory stereotypes  such as racism  sexism  and ableism.   These biases can be perpetuated in various NLP applications  including word embeddings and downstream applications  such as sentiment analysis  job candidate screening  university admissions  and essay grading.   Biases in Word EmbeddingsDevelopers use unsupervised learning to prepare data for NLP models. Specifically  unsupervised models transform raw text data into word embeddings (numerical representations of text data) fed into NLP models. These models analyze massive amounts of text data  such as websites  social media  and books to create vectors that capture a words meaning and its relationship to other words.   However  while searching for hidden patterns in text data  these models are exposed to more than just semantic information  they are subjected to societal biases present in the data. These biases can then be embedded into word embeddings and inherited by supervised models  leading to biased outputs.   For example  sentences in an article might associate words related to doctors  engineers  and scientists mostly with men  while females may be portrayed as nurses  homemakers  or social workers.   Types of Bias in Natural Language Processing ServicesHere are common biases in natural language processing services:   Gender BiasGender bias is a significant and widespread issue in NLP models. Many reports show bias in advanced language models  such as GPT-3  where word embeddings tend to associate men with competency and occupations requiring higher education (doctors  lawyers  CEOs  etc.) in downstream NLP tasks. Whereas  in response to the prompt What gender does a nurse belong to?  it is more likely to output Its female.   Research published in The Artificial Intelligence and Emerging Technology Initiative of The Brookings Institution highlights numerous examples of gender bias in language applications using machine learning. Researchers found that NLP models working with word embeddings picked up biases based on how words are connected in the training data.   For example  words like kitchen and art were more frequently used with the word woman  and words like science and technology appeared in sentences including the word man. Such gender bias embedded in NLP systems leads to biased output.   Racial BiasNLP systems have also displayed racial bias. A 2017 Princeton study discovered that online prejudices against African Americans and the Back community were reflected by model embeddings. As per the study  historically  Black names were more significantly associated with negative words as compared to traditional White names  reflecting real-world prejudices present in training data.   Such racial bias in machine learning extends back even further. The study also mentioned 2004 research that found similar bias in resume assessment done through machine learning algorithms.   Moreover  word embeddings display the most substantial bias for words or phrases representing people with intersectional identities  such as race and gender  relative to other word combinations. For example  the representation of phrases like African American women or Mexican American women can be more negatively biased than just African American or woman alone.   Many AI algorithms creating word embeddings are trained on datasets that reflect the current social order  which can lack diversity and be biased towards certain groups. Due to a lack of diversity  the data used to train word embeddings likely has more information about white men. As a result  other social groups are primarily represented as minorities within the system.   Bias in downstream NLP applications  such as automated resume screening  might not only reflect existing biases but amplify them in society  impacting future generations by limiting their career opportunities.   How to Mitigate Biases in Natural Language Processing   While bias in natural language processing can be handled by debiasing the dataset early on or the model afterward  the ideal approach is to derbies the dataset to prevent the model from learning biased patterns. Here are some effective strategies to derbies natural language processing models:   Data ManipulationAs described earlier  the main reason for bias in natural language processing algorithms is unbalanced original datasets  i.e.  more text associating words related to doctors with male and words nurses with female. With this type of association  the NLP model is more likely to predict male for doctors. To address bias  it is essential to have a balanced dataset where all groups are represented similarly for the model to learn from.   For example  data augmentation algorithms such as SMOTE (Synthetic Minority Oversampling Technique) can be employed to create synthetic data points for the minority group (female doctors) in the dataset. Alternatively  one can choose to remove some data points from the majority group to make the dataset balanced.   Bias Fine-TuningThe bias fine-tuning method leverages the concept of transfer learning. It involves fine-tuning a relatively unbiased pre-trained natural language processing model on a new  more biased dataset. This enables the model to adapt to the specific task requirements of the biased dataset without inheriting biases from that data. Research suggests this method can achieve an accuracy score very similar to the model directly trained on unbiased data.   Data AnnotationData annotation is a crucial step in NLP model development  especially in addressing bias. It involves labeling and categorizing text data to train NLP models. Annotators can flag potentially biased datasets. Biases can include stereotypes  unequal representation of races or genders  or even culturally insensitive language. As a result  developers can take steps to mitigate the bias  such as collecting more balanced data and eliminating biased text data.   Diversity in Developing and Auditing NLP ModelsOther than training data  the bias can emerge from the team developing the model. A study at the 2020 NeurIPS machine learning conference suggested a negative correlation between the level of diversity of the development team and biased NLP models. According to a Brookings Institution study  a diverse AI audit team is essential for the ethical development of machine learning technologies.   A diverse audit group can test the model from various perspectives and help identify and mitigate potential bias throughout the NLP model creation process.   ConclusionThe growing presence of NLP services in our daily lives deepens concerns about bias in their algorithms. These sociotechnical systems absorb human biases and accurately ingest them as they learn from the training language data. This necessitates that development companies take bias mitigation steps to prevent the spread of discrimination further through these technologies.


Making Bayesian Optimization Algorithm Simple for Practical Applications

The Goal of this writing is to show an easy implementation of Bayesian Optimization to solve real-world problems.   Contrary to Machine Learning modeling which the goal is to find a mapping between input and output by utilizing a rather large set of data  in Optimization  defining the exact algorithm inside the Black-Box is not of interest  we do not have the luxury of applying many inputs  Maybe because of the constraints of the process that is too timely or too costly all we are looking for is to find one magical combination of input variables that produces the smallest output and be able to achieve that by examining only a limited number of input values applied to Black-Box.   This problem is prevalent in every discipline  regardless of where you work  you will face this problem where you want to optimize a metric in your process  whether is cost  resources  time to market  quality  reliability  etc.  and in all cases  you have few parameters or knobs you can turn in your process  and you want to find out that magical input values that give you the best-optimized output value with the smallest number of trials.   The situation becomes trickier if black-box output may have some local minimum and maybe one large global minimum  and how can we avoid being trapped in one of those local minimums and missing the largest global minimum?   In this paper  we show how the Bayesian Optimization algorithm  In conjunction with data coming from the field  can work together to discover the optimum point for the process.   You might be sitting at your computer and running a Bayesian Optimization Algorithm  while the physical Black-Box might be sitting in a Lab at some distance. You act as a middleman  talking to both sides. For the Algorithm  we use the SKOPT package of SciKit-learn. You can install this open-source package using the command:   pip install scikit-optimize   Sequential model-based optimization toolbox.pypi.org   The heart of the Algorithm is a Gaussian Process called gp_minimize; for simplicity  Lets call this magical function AI Genie  and You are acting in between this AI-Genie  which is running in your PC  and your physical Black box. The goal of the AI-Genie is to find the minimum output of the black box with as small a number of trials as possible. Also  to make it even simpler  assume that we have only one input in the black box; this process could easily be expanded to a multi-input case. The Picture below shows all the characters in this process:   Here is the actual code:   import numpy as np from skopt import gp_minimize from skopt.space import Real from skopt.utils import use_named_args import matplotlib.pyplot as plt # Define the search space (let's assume we're searching within -100 to 100) search_space = [Real(-100  100  name='X')] # Objective function that interacts with the user @use_named_args(search_space) def objective_function(X): # Print the value of X print(fEnter this value into the black box: X={X}  flush=True) # Ask the user to input the corresponding Y value from the black box Y = float(input(Enter the value returned by the black box (Y): )) # Return the Y value as the result of the objective function return Y # Perform Bayesian Optimization result = gp_minimize(objective_function  search_space  n_calls=15  random_state=0) # Print the result print(fOptimal value found: X = {result.x[0]}) print(fMinimum value of the function: Y = {result.fun}) # Plot the convergence plt.plot(result.func_vals) plt.xlabel('Number of calls') plt.ylabel('Function value') plt.title('Convergence Plot') plt.show()Lets examine the code in more detail:   1- Import required libraries   import numpy as np from skopt import gp_minimize from skopt.space import Real from skopt.utils import use_named_args import matplotlib.pyplot as pltgp_minimize is the main function driving the optimization process   for input parameters to the black box  you can have Integer  Real  and Category. Here  we assume we have just one Real value input   use_name_args is a decorator supplied in SKOPT; its job is to select different values of input parameters and send them to be processed in Black-Box   2- Define search space   # Define the search space (let's assume we're searching within -100 to 100) search_space = [Real(-100  100  name='X')]Offers the system a range of valid values that input can take. For example  here we have one input called X which can take a float value between -100 to 100   3- Black-Box Representation   # Objective function that interacts with the user @use_named_args(search_space) def objective_function(X): # Print the value of X print(fEnter this value into the black box: X={X}  flush=True) # Ask the user to input the corresponding Y value from the black box Y = float(input(Enter the value returned by the black box (Y): )) # Return the Y value as the result of the objective function return YObjective function is the Function representing the Black-Box functionality. The Black-Box is inside the Objective function and receives the input values given to the Objective function from Search Space; the black box accepts that value  it processes the input  and provides the output to objective function  which will then be returned to the optimizing algorithm.   What makes this paper different is that we are acting like a black box inside the Objective function; we get the parameter passed to it  by printing that input value.   Then we pause the program to take that input back to the lab and give it to the physical or virtual black box  get the output of the black box  and then come back to the objective function  which was holding the execution to receive the value we enter as the output coming from the black- box.   and finally  Return the value to the Optimizer and wait for the next input from optimizer.   4- Main Bayesian Optimizer function   # Perform Bayesian Optimization result = gp_minimize(objective_function  search_space  n_calls=15  random_state=0)This is the heart of the algorithm  which we call Ai-Genie; the First parameter for this function is the Objective-function (which holds the black Box inside)  the next parameter is Search_Space  the next parameter is n_calls which the user choose to limit the number of trials  here user is asking the Ai-Genie to provide the minimum value of the output of black box within 15 trials  and last parameter is random_state to initialize the random state.   5- Printing the results   # Print the result print(fOptimal value found: X = {result.x[0]}) print(fMinimum value of the function: Y = {result.fun})This will print the minimum value out of the black box (Y) and the input value (X)  which will get you the minimum output.   Execution   Assume you have set everything and are ready to run the experiment; you have no idea what is inside the black box. You just know for any input you give it  it provides you an out put so lets start the experiment:   1- The First number the optimizer model give you is: 18.568924; the optimizer picks this very first number at random form the range of available input variables.   2- Take this number to the black box  enter it  and wait for the output  The black box returns: 363.373849   3- Take this out  put back to Optimizer  and enter it  wait for Optimizer to provide you with the next number: 68.853150   4- You have finished one round; continue this process till you exhaust the number of trial n_call.   Here X is the number suggested by the Ai-Genie to try on Black Box  and   Y is the output from Black-Box   The final result is given below:   Optimal value found: X = -0.49669415594226507   The minimum value of the function: Y = -0.24998907139506593   Lets plot convergence   # Plot the convergence plt.plot(result.func_vals) plt.xlabel('Number of calls') plt.ylabel('Function value') plt.title('Convergence Plot') plt.show()Notice in a range of -100 to 100  there are an infinite number of float values that Ai-Genie could choose from  but Ai-Genie is so awesome that after testing a few values  it almost knows what the minimum value is after only 10 trials.   Verification   Now that the experiment is concluded  How do I know that the Ai-genie really found the optimum value  and how do I verify it.   In real-world situations  we absolutely do not know what is inside the black box  and we also do not want to know  we are interested just in minimum output  but here just to test the accuracy of the Ai-genie in finding the optimum value  I did not expose this to Ai-genie but I went to black box in the lab  and placed a function that I know inside of it  the function I placed there was :   Y = X**2 + X   We can find the minimum value of this function using Differential equation and set it qual to zero and solve it.   dY/dX = 2X + 1   2X +1 = 0   X = -0.5   Y = -0.25   The values the Bayesian Optimization found without knowing this equation were extremely close  which verifies the power of the algorithm.   This is what makes the Bayesian Optimization algorithm so powerful. We should seriously consider using it more often to find optimal points for any process wherever possible.


Learn Anything with AI and the Feynman Technique

When was the last time you stumbled upon a difficult subject to learn? Or when you spent an hour watching YouTube videos on how to better learn things?   There are countless learning techniques to help you digest complex concepts and feel confident about knowing them by heart. And if youre a student like me who is constantly learning things  you understand the significance of an effective learning approach. One of the simplest one of them is the Feynman Technique.   In this article  I will explain how to apply the Feynman learning method effectively  and how you can use Artificial Intelligence to fill in the gaps of your knowledge.   By the end  you will be able to use ChatGPT to break down complex concepts and master them intuitively and effortlessly in four easy steps!   What is The Feynman Technique?Richard Feynman was an American theoretical physicist. As part of the Manhattan Project  He played a crucial role in the development of the atomic bomb during World War II. In 1965  he won the Nobel Prize in Physics for his work on quantum electrodynamics. But beyond all that  he was a popular teacher and author of famous books.   Despite all the impressive achievements  Feynman didnt believe himself to be intellectually special  but rather an ordinary person who could commit himself to studying hard.   I was an ordinary person who studied hard  theres no miracle people. Theres no talent or special miracle to study quantum mechanics that comes without practice and reading and learning and studying.  Richard Feynman [1]   Now the Feynman Technique is not directly devised by Feynman but associated with him. Nevertheless  it is inspired by how Feynman believed a subject must be studied.   I couldnt reduce it to the freshman level. That means we dont really understand it.  Richard Feynman [2]   Feynmans TechniqueFeynman was famous for his ability to explain complex physical concepts in an intuitive and digestible fashion. He believed that you can only claim you have understood a concept  if you can explain it understandably  to someone who does not have any prior knowledge about it. Nobody could say it better than Feynman himself    When we speak without jargon  it frees us from hiding behind knowledge we dont have. Big words and fluffy business speak cripples us from getting to the point and passing knowledge to others.   Feynmans technique for learning a topic can be broken down into these four simple steps:   Teach the concept: The most effective method to understand something is by teaching it. Whether you want to imagine teaching the concept to someone else  yourself  or an imaginary child  you must assume the other person knows nothing about the subject. So dont hide behind and from big words.Identify gaps: Go through what you have been teaching. From the other persons point of view  try to identify parts of your explanation that are missing  need more work  or are simply not understandable enough.Refine: Using the feedback of the last step  iteratively refine your explanation until you are happy with it.Tell a Story: Now that you have the foundation  fortify it with examples  illustrations  and diagrams. Make your explanation so fluent that you can convey it through a story that is easy to the ears and entertaining to follow.AI + Feynman Technique = U+1F525You're falling behind if youre not using AI to enhance your learning process. In this section  I will introduce a very simple method to couple AI with the Feynman Technique by prompting ChatGPT. As an example  I will use vector database as a machine learning concept I would like to learn.   Step 1: I will read about the topic and study a basic understanding of it. Then  I developed a simplified explanation of what a vector database is. Lets say I come up with the following explanation:   Imagine having a library of books. You can add books to the library and retrieve them quickly by using their name or any other indexing method. A vector database is also a library  but instead of books  stores vectors. A vector can be thought of  as a list of numbers that represent an image  audio  or any sort of data. Once we convert the data into vectors  by using various machine learning techniques  we can store the vectors and retrieve them efficiently using some indexing method.Alternatively  we could ask ChatGPT to provide us with an explanation of the concept if we have trouble writing it ourselves. We can use this prompt to ask ChatGPT:   Explain [concept] to me as if I am a complete beginner with no prior knowledge.   Step 2: Once we have an initial definition of our concept  its time to identify how we can improve upon it. We can ask ChatGPT to act as our audience and give us feedback. The prompt for this:   Identify potential gaps or unclear points in my explanation of [concept]: [explanation]   By using this prompt and my explanation of vector database  I get a list of suggestions on which parts of it could have knowledge gaps or more clarification:   Expand on Vectors: Explain how vectors are derived from data  using a specific example to illustrate the concept. Detail Conversion Techniques: Mention specific machine learning techniques  like neural networks or NLP  used to convert data into vectors. Clarify Indexing and Retrieval: Discuss specialized algorithms used in vector databases  such as approximate nearest neighbor search  and how they differ from traditional indexing methods. Include Use Cases: Add examples of practical applications like multimedia retrieval and real-time recommendation systems to show the benefits of vector databases. Discuss Scalability and Performance: Mention how vector databases are optimized for handling large datasets and high-performance needs. Enhance Comparison with Traditional Databases: Extend your library analogy to compare the search processes in vector and traditional databases  emphasizing efficiency and accuracy in finding similar items.Step 3: Repeat the previous steps until you reach a concise and complete definition of your concept that you are happy with.   Step 4: Expand your explanation by adding analogies and diagrams. Using gpt-4o you can also retrieve diagrams and images to help you get a visual understanding.   I will use two separate prompts  one for analogies:   Add two analogies to help develop a more understandable explanation of [concept].   Another prompt for drawing a diagram of the concept:   draw me diagrams to help understand the concept of [concept]   ChatGPT will continue to create a diagram for a full understanding of vector databases:   U+2622WARNING: It is crucial to have in mind that AI hallucinates! This means that it tends to make up information that doesnt exist. To make matters worse  AI sounds confident in making up these mistakes  so unless you already have some prior knowledge about a topic  fully handing the steering wheel to AI needs caution!   Thanks for reading!~ Hesam   [1] Richard Feynman Thinking Part 1 of 2   [2] Feynmans Lost Lecture


But What Is Inside an AI Accelerator?

Heterogeneous computing refers to machines with more than one kind of computing core. The computing cores can be CPUs  GPUs  TPUs  and many other accelerators that are being developed every day. These specialized cores can also be called ASIC an abbreviation for Application-Specific Integrated Circuit.   This is how ARM defines ASIC    An application-specific integrated circuit is an integrated circuit (IC) thats custom-designed for a particular task or application. Unlike FPGA boards that can be programmed to meet a variety of use case requirements after manufacturing  ASIC designs are tailored early in the design process to address specific needs.   Since the release of ChatGPT and the subsequent release of other large language models (LLM)  there has been a growing demand for computing power that is required to train these models (with billions of parameters) and also generate results  which is called inferencing. This is precisely where AI Accelerators come to the rescue!   An overview of what lies ahead in this article   In this article  I will go over a small introduction to AI accelerators and how they differ from CPUs and GPUs. Then I will dive into systolic array architecture and how it works! I also peek inside the Google TPU and end the article with possible future research directions.   Introduction to AI AcceleratorsAI accelerators are specialized hardware designed to enhance the performance of artificial intelligence (AI) tasks  particularly in machine learning and deep learning. These accelerators are designed to perform large-scale parallel computations (read matrix multiplications) as required by many deep learning models efficiently as compared to traditional CPUs.   Some key characteristics that differentiate AI Accelerators from CPUs and GPUs are:   They are a type of ASIC specifically designed for deep learning workloads. In contrast  CPUs and GPUs can also be used for general-purpose programming and rendering graphics  respectively. NVIDIA GPUs in fact  started out as ASIC for handling computer graphics-related operations and then transitioned into being used in scientific computing (with the help of CUDA). Sometime later  around 2015  the focus of CUDA transitioned towards supporting neural networks.Massive parallel processing power  GPUs and accelerators are designed to execute many operations in parallel (high throughput)  whereas CPUs are designed to perform sequential operations in the shortest time (low latency). Accelerators are meant to offload deep learning workloads from CPUs so as to perform these operations more efficiently.Systolic ArraysSystolic array is a simple and energy-efficient architecture for accelerating general matrix multiplication (GEMM) operations in hardware. They provide an alternative way to implement these operations and support parallel data streaming to improve memory access and promote data reuse. This architecture forms the basis of many commercial accelerator offerings like the Google TPU (tensor processing unit)  Intel NPU (neural processing unit)  IBM AIU  etc.   These arrays comprise MAC (multiply-and-accumulate) units that perform the actual operations. Serving the MAC units are the row and column SRAM buffers that feed these units with data. Each MAC unit will save the incoming data in an internal register and then forward the same data to the outgoing connection in the next cycle.   This behavior results in significant savings in SRAM read requests and can exploit data reuse opportunities. For example  filter weights are something that remains stationary during a convolution operation as the filter map is convolved over the image. This can be exploited by storing the weights in the MAC array whereas the row buffer loads in the different windows of the input image. This reduces the read requests to load the weights  hence freeing up bandwidth to read from off-chip memory sources like DRAM or HBMs.   There are different techniques to exploit data reuse  which are referred to as dataflow or mapping schemes discussed in the next section.   Data Flow TechniquesAlthough there are no hard and fast rules to specify what kind of mapping is to be used with a systolic array architecture  here I will discuss one of the three strategies as specified in the Scale-Sim paper. The three strategies are named Output Stationary (OS)  Weight Stationary (WS)  and Input Stationary (IS). The word stationary here depicts what part of the computation spends the most amount of time being stored in the systolic array.   The output stationary dataflow is depicted in the figure above. Output stationary means that each MAC unit will be responsible for calculating the output pixel. All the required operands are fed from the left and top edges of the systolic array. Each row (IFMAP) consists of elements of one convolution window and one column (FILTER) entering from the top represents the unrolled filter. Elements of one row and one column are multiplied and accumulated to calculate one pixel of the output feature map (OFMAP).   Timing Model for a Systolic ArrayHere we try to calculate the number of cycles that a systolic array will take to perform a matrix multiplication. Here we assume that there are no stalls during the operation due to memory bandwidth (make sure that SRAM buffers are filled with data to perform the compute) and also assume that we have unlimited MAC units available to perform the required computation.   Sr  Sc are the dimensions of the systolic array and in this case  is equivalent to the number of rows and columns of the IFMAP and FILTER respectively. T is the temporal dimension which in the case of the output stationary represents the convolution window size.   As described by the figure above  we can conclude that the number of cycles for the systolic array to perform a matrix multiplication is:   Obviously  in the real world  we do not have unlimited MACs. In that case  we divide the workload by the number of available MAC units and therefore get the following expression for timing:   Here  we assume that R and C are the actual dimensions of the systolic array and Sr and Sc are the required dimensions. To decrease this time  we can increase the number of MAC units  a process we can call scaling up. Another approach is to have multiple MAC array units that perform the compute in parallel  which can be called scaling out. This further reduces the time needed to complete the operation.   A look inside Google TPUOriginsBack in 2013  a projection at Google showed that if people searched using voice even for 3 minutes a day  it would result in doubling the computing demand of Googles datacenters. Speech recognition models that used DNN were very expensive to perform inference using traditional CPUs. Therefore  they started working on a custom ASIC (application-specific integrated circuit) that would perform inference efficiently. The goal was 10x performance over GPUs. The outcome of this effort was the Google Tensor Processing Unit. Google TPU was based on the systolic array architecture.   TPU v1As you are now aware systolic array-based AI accelerators are composed of MAC units. Googles original TPU implementation consisted of 256x256 MAC units (see Matrix Multiply Unit in the figure above) that could perform 8-bit multiply-and-adds on signed or unsigned integers. The 16-bit products were then collected in 4 MiB of 32-bit Accumulators below the matrix unit. Then there are other components like the activation pipeline that could perform activation functions on the resulting matrix.   For more details about the Google TPU that was released in 2017 read this very interesting paper where they discuss in detail the TPUs design and performance!   In-datacenter performance analysis of a tensor processing unit U+007C IEEE Conference Publication U+007C IEEE Xplore   TPU v2 and v3Improving upon the design of TPU v1 Google released the specifications of TPU v2 and v3 as well with some major changes:   Interconnect  A critical element of any chip design is the interconnect which decides how fast is the inter-chip communication. An on-device switch called Interconnect Router (see above figure) provides deadlock-free routing. It enables a 2D torus topology of interconnect.Memory  A major performance bottleneck in TPU v1 was the limited memory bandwidth of DRAM. This problem was somewhat solved using the HBM (High Bandwidth Memory) DRAM in TPU v2. It offers 20 times the bandwidth of TPU v1 by using an interposer substrate that connects the TPU v2 chip via thirty-two 128-bit buses to 4-stacks of DRAM chips.Multiple smaller MXU units per chip  While TPUv1 featured a MXU of the size 256x256  it was reduced to 128x128 in TPUv2 onwards and has multiple MXUs per chip. Larger MXUs require more memory bandwidth for optimal chip utilization. Google analyzed that convolutional model utilization ranged between 37%-48% for 128x128 MXUs  which was 1.6x of a single 256x256 MXU (22%-30%). The reason that Google has come up with this is that some convolutions are naturally smaller than 256x256  which leaves parts of the MXU unused.For more details regarding Google TPU v2 and v3:   A Domain Specific Supercomputer for Training Deep Neural Networks U+007C ACM   AI and Memory WallThe amount of computing needed to train modern deep learning models and perform inference using them is growing at a large rate. This trend prompted research into AI accelerators with a focus on increasing computing power. This has been achieved sometimes at the expense of neglecting memory hierarchies and bandwidth thus creating a memory bottleneck. In this section  I have briefly summarized what this very interesting paper [Gholami et al.  2024] talks about and which points toward future research avenues in the realm of AI accelerators.   But what is a memory wall?   Memory wall refers to the problem where the compute is faster than the rate at which data can be fetched from off-chip DRAM which limits the overall compute that can be performed. The time to complete an operation is dependent both on the speed of performing compute and also on how fast the data can be fed to the arithmetic units of hardware.   As can be seen in the graph above  the peak compute has increased 60000x in the last 20 years  whereas the DRAM and interconnect bandwidth have increased only 100x and 30x  respectively. This huge deficit results in aggravating the problem of memory wall especially with growing model sizes.   As depicted in figure (a) above the number of parameters in the SOTA transformer models has increased at a rate of 410x every two years  whereas the AI accelerator memory capacity (green dots) has only been scaled at a rate of 2x every 2 years. Figure (b) depicts the amount of compute  measured in Peta FLOPs  needed to train SOTA models for different computer vision (CV)  natural language processing (NLP)  and Speech models  along with the different scaling of Transformer models (750x/2yrs).   This problem opens up many research avenues where progress can be made. Techniques like quantization and model pruning are being actively investigated to reduce model size. One of the major breakthroughs in AI accelerators has been the successful adoption of half-precision (FP 16) instead of single precision enabling a 10x increase in hardware compute capability. Another possible solution that the author proposes worth investigating is revisiting the organization of the cache hierarchy of AI Accelerators that has been simplified to prioritize computing power.   Do check out the paper by the author for a more detailed analysis and discussion on this topic: [2403.14123] AI and Memory Wall (arxiv.org)   Further ReadingDNN Accelerator Architecture  SIMD or Systolic? U+007C SIGARCHArchitecting Chips For High-Performance ComputingHow To Build A Better Blackwell GPU Than NVIDIA DidExtending Dataflow Techniques from Dense to Sparse Accelerators U+007C SIGARCHReferencesJouppi  N. P.  Young  C.  Patil  N.  Patterson  D.  Agrawal  G.  Bajwa  R.   & Yoon  D. H. (2017  June). In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture (pp. 112).Jouppi  N. P.  Yoon  D. H.  Kurian  G.  Li  S.  Patil  N.  Laudon  J.   & Patterson  D. (2020). A domain-specific supercomputer for training deep neural networks. Communications of the ACM  63(7)  6778.Gholami  A.  Yao  Z.  Kim  S.  Hooper  C.  Mahoney  M. W.  & Keutzer  K. (2024). AI and memory wall. IEEE Micro.Samajdar  A.  Joseph  J. M.  Zhu  Y.  Whatmough  P.  Mattina  M.  & Krishna  T. (2020  August). A systematic methodology for characterizing scalability of dnn accelerators using scale-sim. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) (pp. 5868). IEEE.


Month in 4 Papers (June 2023)

Advancing Language Models through Efficient Training and Alignment Techniques.      This series of posts is designed to bring you the newest findings and developments in the NLP field. Ill delve into four significant research papers each month  offering a comprehensive summary. Be sure to visit my blog regularly or subscribe to my newsletter for monthly updates. Lets dive in!   U+1F4DD Better & Faster Large Language Models via Multi-token Prediction [paper]   This paper proposes an approach where multiple tokens are predicted using multiple heads  shifting from the conventional method of predicting only the next token. The method uses a shared model (called trunk) containing 13 billion parameters. During training  tokens are processed individually  with their losses computed and aggregated before the backward pass and weight updates are done. This ensures that memory usage will not grow.   During the inference phase  the model can generate output tokens sequentially as previously done or leverage the proposed method to accelerate the inference process by a factor of three.   This method proved most effective on coding benchmarks like HumanEval and MBPP. Their thorough analysis indicates that the effectiveness of this method becomes more apparent as the scale increases. Moreover  experimenting with various numbers of heads revealed that predicting four tokens in advance yielded the greatest result improvement. They demonstrated a 12% enhancement in HumanEval and a 17% increase in problem-solving rates on MBPP. Although they applied the approach to tasks like Q&A and summarization  it didnt boost performance but can significantly speed up inference processing.   Other researchers have explored multi-token prediction techniques; this paper stands out for its innovative approach and comprehensive model analysis  making it a great read. However  it would have been nice if they had released the code  too.   Extended LSTMU+1F4DD xLSTM: Extended Long Short-Term Memory [paper]   The author of LSTM released the idea of xLSTM to overcome the limitations of the original architecture. One of the important aspects was the lack of parallelization  which slowed the network during training/inference. The two novelties of this paper are the use of exponential gating (instead of Sigmoid) and the replacement of scalar memory with Matrix memory. These ideas  amongst others  led to the creation of the sLSTM and mLSTM memory cells. Stacking the two mentioned components with a residual connection forms an xLSTM component  and multiple xLSTM components can be layered to create the xLSTM architecture.   The resulting model has parallel processing capabilities during both training and inference. The network benefits from increased memory capacity and enhanced memory updating efficiency. Notably  it incorporates an attention-like mechanism using key/value/query vectors within its components. The model achieves faster performance and uses fewer computational resources than the transformer architecture while slightly outperforming or matching transformer-based models in text generation and classification.   Unlike what I thought when I saw this paper  Its more like a transformer network rather than a traditional LSTM. The only common element in the new architecture is the idea of gated design!   DPO vs PPOU+1F4DD Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study [paper]   Since the release of the DPO paper  theres been a lot of buzz about whether the DPO approach  which is notably simpler than PPO  performs at the same level. Companies like OpenAI use Reinforcement Learning (RL) to train models such as ChatGPT  whereas many open-source/academic projects do DPO. The advantage of not needing to train a reward model is that it is more feasible to train models with fewer resources and fewer trials.   Experiments were conducted to evaluate the performance of LLMs tuned with DPO and PPO. The models were tested on the HH-RLHF dialogue task and two coding tasks. The results demonstrated that PPO consistently improves the models performance on complex tasks such as coding. They also discovered that using iterative DPO  which involves generating additional data with the newly trained rewards model during the tuning process  is more effective. However  PPO still outperforms DPO and achieves state-of-the-art results on challenging coding tasks.   Lastly  the ablation study highlights the crucial elements for the success of PPO training: normalizing advantages  using large batch sizes  and updating the reference model parameters with an exponential moving average.   No Attention?U+1F4DD Pretraining Without Attention [paper]   The idea is to explore if we can match the performance of the transformer-based models without an attention mechanism. They propose an architecture based on the combination of State-Space models (SSMs) and multiplicative gating.   They replaced the attention-based routing with the State-Space models. (High-level overview coming details not necessary for now!) These models describe a systems behaviour by linking unobservable variables to controllable inputs and measurable outputs. The model offers a method to achieve long-range dependencies similar to RNNs with the training speed of CNNs. Interestingly  they achieved comparable accuracy to BERT on the GLUE benchmark by simply matching the number of parameters!   The BiGS model does not exhibit quadratic complexity in relation to the length seen in transformers; instead  its complexity is linear at 2L. The paper suggests that this model may be the first to rival transformers without using attention. This fascinating research indicates that the transformer architecture isnt inherently unique or special. There may be other architectures using the same components but arranged differently that perform similarly yet more efficiently.   Maybe we should focus on finding different architectures and techniques at the same time that we are scaling the transformer to jizilion parameters :)   I send out a monthly newsletter for NLP nerds. Consider subscribing if you like to stay up-to-date on the latest developments in Natural Language Processing. Read more and subscribe  join the cool kids club and sign up now!   Final Words Please let me know what you think of this series. I would like to hear your feedback. What parts were interesting? What sections did you not like? What was missing  and do you want to see more of it in the future? Please reach out to me at nlpiation@gmail.com.


AI Trends on TED

Introduction: The AI Zeitgeist Through the TED LensIf you are like me  you turn to TED videos to satisfy your curiosity or to be educated on innovative ideas. In recent years  I have regularly watched their Artificial Intelligence videos to learn more about AIs capabilities  potential  and risks. Almost every week  I notice a new TED AI video on my YouTube homepage  which inspired me to do some digging. There are over 550 AI-themed TED videos dating back to 2007!   The dataset  interactive app and YouTube playlist at the end of the article.   As I started to explore TEDs rich library of content  it dawned on me that these conversations are a window into how AI technology and adoption is evolving. With this in mind I started my most extensive data analysis project to date  to give some structure and track the TED trends. I used the YouTube videos as my source  a little LLM  and a lot of Python to build a knowledge graph and started analyzing. This article is about the analysis  not the graph  but follow me for a future article about the build.   The Evolution of AI on TEDMy first step in the analysis was to better understand the video publishing trends over time. Using this dataset as a foundation I started to investigate what was the story behind these trends.   Early Days: Visionaries and Pioneers (20072015)In the mid-2000s  when AI was still a niche topic for most  TED was already featuring talks by visionaries like Ray Kurzweil  known for his work on The Singularity  and Jeff Hawkins of Palm Computing fame. IBMs Jeopardy playing Watson  launching in 2011  was the biggest AI story of the time. During this period  AI discussions were sporadic  appearing occasionally but not consistently every year.   It is also notable that TEDx events were already in the thousands by 2012 (source: Forbes)  so either the content was not focused on AI  or these videos were not published on YouTube (or are now archived)   The Tipping Point (20162017)Based on the dataset  a shift began in 20162017  marked by an increase in AI coverage. DeepMinds AlphaGo was mastering Go  not by memorization  but by creating its own new strategies that beat the Go masters  such as its victory over world champion Lee Sedol in 2016.   At the same time TEDx events were spreading (with over 100 000 talks by 2017  source: TED Blog) and the topic of AI intrigued many of the new community-based presenters.   The Deep Learning Boom (20172019)The increase in AI-related TED talks during 20172019 resulted from several factors converging at once. This period saw advances in deep learning and neural networks research and at the same time  companies/venture capitalists increased their investments in AI startups. Data Science became a popular career choice  and Big Data was a hot topic. AI technologies also reached the top of Gartners Hype Cycle for Emerging Technologies  reflecting high public interest and high expectations.   These factors  tech progress  more funding  growing expertise  and public excitement  led to more AI discussions at TED talks. People were seeing how AI would impact different aspects of society and industry. TED became a forum for exploring this AI shift as it happened.   The Pandemic Interlude (20202021)During 20202021  much of the focus on TEDs main channel shifted to healthcare  remote work  and the social impacts of the COVID-19 pandemic. AI was not the main topic but was an undercurrent in the discussions about technological solutions to pandemic-related challenges.   The ChatGPT Era (Late 2022-Present)ChatGPT-3s release in late 2022 sparked renewed interest in AI  especially in Large Language Models (LLMs). Throughout 2023 and 2024  AI and LLMs have taken center stage at TED. Presenters have covered a wide range of topics  from the technologys capabilities and opportunities to its societal impacts and potential risks.   And to no ones surprise  TED is not alone. A snapshot from Google Trends shows the impact of AI on search is even more dramatic. Interest in AI experienced a parabolic shift and is only now stabilizing at levels 10x what they were before ChatGPT.   The volume and publishing cadence of the videos tell part of the story  now lets see what we can extrapolate from the videos themselves.   What Can We Learn from the Video DataNext we will dig into the content of this collection of YouTube transcripts and metadata. My analysis involved extracting key concepts (topics  people  organizations  etc.) as well as categorizing the videos to build a knowledge graph. With this we can learn about the categories  people and organizations that dominate the TED AI video and also provide insights into the general zeitgeist of Artificial Intelligence.   Key CategoriesAI is a general-purpose technology  much like electricity or the Internet  with the potential to achieve significant results across a wide range of applications. This diversity is reflected in the categories and topics in the video dataset. AI has the potential to impact various areas of life  including business  society  healthcare  education  work  art  entertainment  and more. Alongside these emerging applications  we also see videos addressing a broad set of concerns  including ethics  governance  safety  and security.   In terms of distribution  the TED catalog is actually very balanced across these two extremes. Applying AI to business and industry is a major focus of the TED catalog  with 126 videos dedicated to this category. However  this focus is balanced by a significant number of videos addressing societal impacts (113) and AI ethics and governance (99). The pattern continues with substantial categories focused on healthcare (63) and education (55)  balanced by concerns about the future of work (36). As we move into the smaller categories  this pattern of balance persists. Overall  about 55% of the videos primarily focus on opportunity topics  and 45% focus on more risk-related topics.   The fact that opportunities and risks weigh evenly in TED presentations mirrors the dilemma we face as a society  what will it cost us to embrace the potential of AI?   Influential PeopleNow lets move on to what can be learned about AI by examining the individuals mentioned in these TED videos. Key individuals frequently mentioned in the videos fall into three categories:   Technical Thought Leaders: Known for their pioneering contributions and thought leadership in AI (e.g.  Alan Turing  Stephen Hawking  Ray Kurzweil  Marvin Minsky).Business Leaders: Visionaries in the business world who have significantly influenced the adoption and application of AI/Technology (e.g.  Elon Musk  Bill Gates  Mark Zuckerberg  Steve Jobs).Expert Reference Points: Masters in their fields who have been profoundly impacted by AI advancements (e.g.  Garry Kasparov in chess  Lee Sedol in Go  Michelangelo in art).While many of these names are well-known  there were a few that I had to research  with the larger list feeling almost like a whos who in AI quiz.   More so than the abstract trends and concepts  understanding the individuals in AI helps to give a broader context to what we see in the video library. This AI moment is historical  and these individuals will be an important part of that history.   Leading OrganizationsOrganizations also play an important role  and while I dont think the list of most referenced organizations will surprise anyone  it does highlight key shifts over the 17 years of TED videos.   Google is mentioned almost twice as often as the next organization  even considering their DeepMind acquisition as a separate entity.OpenAI has rapidly gained prominence  despite being a relative newcomer.MIT and Stanford are the leading academic institutions for AI research and development.IBM  Amazon  and Meta have been minimally referenced in this latest LLM wave  and over 80% of their mentions happened before 2022.Organizations have much more inertia than individuals  and I think we will continue to see Google  Microsoft  MIT  Amazon  etc.  for many more years. That is not to say there will not be upstarts like OpenAI   but it is far more likely their star will fade or they get consumed (e.g. DeepMinds acquisition by Google). For this trend  our 17 year window might not be enough.   ConclusionThese TED Talks serve as a window into the AI revolution  reflecting its journey from a niche subject to a transformative force in our society. This analysis leverages video content to provide insight into AI technology trends  societal opportunities and risks  and the individuals and companies driving its emergence.   As AI continues to evolve  TED videos will remain valuable resources for understanding its potential  challenges  and the critical conversations surrounding its development and implementation. While these individual presenters and videos are incredibly powerful on their own  analyzing them in aggregate reveals trends that enhance our broader understanding.   The story of AI is still in its early chapters  and it will be fascinating to see how these trends evolve and what new topics emerge in this dynamic field.   Resources to Explore FurtherPlaylist of YouTube VideosVideo DatasetApp to Search and Explore VideosData Call-OutsThe goal of the dataset is to be directional and insightful. I believe it achieves this  but alas it is not perfect.   The video playlist contains videos published through May 2024. As a result many of the charts have full year data for other years and partial for 2024.This playlist was manually generated  by myself. I may have made errors or applied judgment on what to include inconsistently I did my best.There are other TED Videos published that are not on the YouTube channel  so this playlist and dataset is incomplete.The playlist includes all of the TED channels in this analysis. By including all of these channels  we can get a broader cross-section of what people are interested in sharing and discussing. The main TED channel features videos from the official TED conference and TEDx videos that have been promoted to the main channel. TEDx actually has many more videos  as it comes from numerous community-organized events. There is also a TED-Ed channel  which focuses on educational content. Lastly a seemingly inactive TED Institute channel that was more corporate-focused.The extractions and category assignments were done with OpenAI ChatGPT-4o. There can be inconsistencies and errors.While not a focus of this analysis  the YouTube stats (Views  Likes  Comment  etc) were updated at the beginning of July 2024. There is an inconsistency with YouTube metrics in that a video published months or years before another video has had more time to accumulate  the Views  Likes  and Comments. Since the last video was added at the end of May  there was at least a one-month period for the video statistics to accumulateMethodologyBelow is the general methodology I used in conducting this analysis. I plan to do a separate article on the process and techniques in the future (follow me to learn more).   Identified YouTube and established a playlist of relevant videos thru ~June 1  2024Used APIs and Python to gather both YouTube metadata and transcripts.Processed the data in a Python notebook  including transcript summarization  concept extraction  and categorization. This was done with the OpenAI API (i.e. LLMs).The results were stored in a knowledge graph comprising over 3 500 nodes and 11 000 relationships.Manually reviewed the captured nodes and relationships to remove issues/errors  and merge similar concepts (Stanford vs Stanford University  etc).Created datasets useful for analysis (e.g.  video count by year/channel  video count by person  etc) then created visualizations.As a side effort I loaded this knowledge graph data into a JSON file for the web app.


A Practical Guide to Building GPT-2 with PyTorch (Part 2)

This is the second part of the GPT-2 from scratch project. If you havent read the first part yet  I highly recommend getting familiar with the language model basics before continuing.   Build and Train GPT-2 (Part 1)Final Loss:   4. Implement GPT-2 architectureIn this section  we will add the GPT-2 parts one by one and then train & evaluate how the model performs in each stage. Heres how it goes:   a. Positional Encoding + Fully Connected Layer (NN)   b. (Masked) Self-Attention + Normalization   c. (Masked) Multi-Head Attention   d. Multiple GPT Decoder Blocks   e. Improving Tokenizer   f. Final GPT-2 Training   To recall from previous part  our model looks like below:   Code:   import torch.nn as nn import torch.nn.functional as F # used to define size of embeddings d_model = vocab_size class GPT(nn.Module): def __init__(self  vocab_size  d_model): super().__init__() self.wte = nn.Embedding(vocab_size  d_model) # word token embeddings def forward(self  inputs  targets = None): logits = self.wte(inputs) # dim -> batch_size  sequence_length  d_model loss = None if targets != None: batch_size  sequence_length  d_model = logits.shape # to calculate loss for all token embeddings in a batch # kind of a requirement for cross_entropy logits = logits.view(batch_size * sequence_length  d_model) targets = targets.view(batch_size * sequence_length) loss = F.cross_entropy(logits  targets) return logits  loss def generate(self  inputs  max_new_tokens): # this will store the model outputs along with the initial input sequence # make a copy so that it doesn't interfare with model for _ in range(max_new_tokens): # we only pass targets on training to calculate loss logits  _ = self(inputs) # for all the batches  get the embeds for last predicted sequence logits = logits[:  -1  :] probs = F.softmax(logits  dim=1) # get the probable token based on the input probs idx_next = torch.multinomial(probs  num_samples=1) inputs = torch.cat([inputs  idx_next]  dim=1) # as the inputs has all model outputs + initial inputs  we can use it as final output return [decode(out.tolist()) for out in inputs] m = GPT(vocab_size=vocab_size  d_model=d_model).to(device)Now  lets add Positional Encoding into our model:   The Output Embedding (in our case its the input embedding wte) is added with Positional Encoding and then passed into the further network. To understand what PE is  lets recall token embedding which stores d_model dimension of vector for each character in our vocabulary. It represents different properties of the character based on how and where it appeared while training.   Similar to this  the Positional Encoding stores the order/positional signal of every character in the context_length. It is only calculated once using sine and cosine functions and doesnt need training. This means the positional vector of each character in the sequence will be same for all the data in training set.   So  when we add them both together we get the property + position of the characters in a sequence which will help model learn better.   I will only show the things I added in the code blocks so that you can add them accordingly. If theres any modifications  I will change lines to + for added lines and  for removed lines.   This is how you can add PE to the model:   # define our PE Class class PositionalEncoding(nn.Module): def __init__(self  context_length  d_model) -> None: super().__init__() # Create a matrix of shape (context_length  d_model) to store the positional encodings pe = torch.zeros(context_length  d_model) # Create a vector with positions [0  1  2  ...  context_length-1] of shape (context_length  1) position = torch.arange(0  context_length  dtype=torch.float).unsqueeze(1) # Create a vector with the divisor terms based on the dimension div_term = torch.exp(torch.arange(0  d_model  2).float() * (-math.log(10000.0) / d_model)) # Compute the positional encodings using sine and cosine functions pe[:  0::2] = torch.sin(position * div_term) pe[:  1::2] = torch.cos(position * div_term) pe = pe.unsqueeze(0) # Shape: (1  context_length  d_model) # Register pe as a buffer  so it is not considered a parameter but is part of the module's state self.register_buffer('pe'  pe) def forward(self  x: torch.Tensor) -> torch.Tensor: # Add the positional encodings to the input embeddings return x + self.pe[: :x.size(1)  :] class GPT(nn.Module): def __init__(self  vocab_size  d_model): ... # initialize positional encodings self.wpe = PositionalEncoding(context_length  d_model) def forward(self  inputs  targets = None): logits = self.wte(inputs) # pass logits to the PE logits = self.wpe(logits) ... return logits  loss ...Now  if you try to train the model and generate a sequence  you would get an error like below:   This basically means we tried generating 1000 tokens one by one and passing previous n tokens to model for getting next token. But now that we have a PositionalEmbedding layer  it only expects token of size less than or equal to the context_length which is 256 in our case.   Lets modify our generate function to accommodate the context_length:   def generate(self  inputs  max_new_tokens): # this will store the model outputs along with the initial input sequence # make a copy so that it doesn't interfare with model output = inputs.clone() for _ in range(max_new_tokens): current_seq_length = inputs.size(1) # Truncate inputs if it exceeds context_length if current_seq_length > context_length: inputs = inputs[:  -context_length:] ... output = torch.cat([output  idx_next]  dim=1) return [decode(out.tolist()) for out in output]We can already train our model and observe improvements  but before jumping into that  lets add one more layer of mapping. Recall how we are currently obtaining different representations of characters and feeding them into the model. How beneficial would it be if we had additional networks to combine this information and learn more complex representations of the embedding?   That would be the Fully Connected Networks. Lets add PyTorchs Linear Layer in our model:   Code:   class GPT(nn.Module): def __init__(self  vocab_size  d_model): ... self.fcn = nn.Sequential( nn.Linear(d_model  4 * d_model)  nn.GELU()  nn.Linear(4 * d_model  d_model) ) def forward(self  inputs  targets = None): ... logits = self.fcn(logits) ... return logits  lossThats it  simple as that !! Now lets train and evaluate the performance of our model. Im setting the epochs to 5000 and learning rate to 1e-3 for this run.   Maybe not much of an improvement but its now starting to form correct words which it is learning through the position of the characters. Lets keep going shall we?   b. (Masked) Self-Attention + NormalizationU+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525U+1F525   ATTENTION HERE   This is the most interesting part of a transformer model: Self-Attention. To make this concept more clear  refer to the visuals from Jay Alammar. In simple terms  Self-Attention defines which next token the model should pay more attention to given the current and previous n tokens.   It does this by assigning scores to the embedding of each character (in our case) and combines them based on different contexts using Queries  Keys  and Values.   Now enough of the theory  lets get into coding:   Heres how you can add Self-Attention to your model:   class SelfAttention(nn.Module): def __init__(self  d_model: int): super().__init__() self.query = nn.Linear(d_model  d_model) self.key = nn.Linear(d_model  d_model) self.value = nn.Linear(d_model  d_model) self.fc_out = nn.Linear(d_model  d_model) self.dropout = nn.Dropout(0.2) def forward(self  inputs: torch.Tensor): B  seq_length  d_model = inputs.shape # Project the input embeddings into Q  K  and V Q = self.query(inputs) K = self.key(inputs) V = self.value(inputs) # Compute attention scores attention_scores = torch.matmul(Q  K.transpose(-2  -1)) # Apply mask to prevent attention to future tokens mask = torch.triu(torch.ones(seq_length  seq_length)  diagonal=1).bool().to(inputs.device) attention_scores = attention_scores.masked_fill(mask  float('-inf')) attention_weights = torch.softmax(attention_scores  dim=-1) # Compute the weighted sum of the values attention_output = torch.matmul(attention_weights  V) # Apply the final linear transformation out = self.fc_out(attention_output) return out class GPT(nn.Module): def __init__(self  vocab_size  d_model): ... self.att = SelfAttention(d_model) def forward(self  inputs  targets = None): ... logits = self.att(logits) logits = self.fcn(logits) ... return logits  lossSimple as that. Now lets train the model and see the outcome:   U+1F60DWOW. Is is only me or do you also think that the model is now starting to understand a lot of word representation and how its put together in a song? Thats pretty impressive. Wait till this layer gets Multi-Head.   Normalization   One thing you may notice if you are training your model along with me is that the losses are decreasing very quickly  and the model is starting to overfit the data. This can happen because the model is becoming too large relative to our limited training data.   To mitigate this  lets add few LayerNorm and Dropout layers to balance out the learning.   Code:   class GPT(nn.Module): def __init__(self  vocab_size  d_model): ... self.ln1 = nn.LayerNorm(d_model) self.ln2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(0.2) def forward(self  inputs  targets = None): ... logits = self.wte(inputs) logits = self.wpe(logits) att_logits = self.att(logits) adn_logits = self.ln1(logits + att_logits) logits = self.dropout(adn_logits) logits = self.fcn(logits) logits = self.ln2(logits + adn_logits) ... return logits  loss ...This will help us train model for a longer period without over-fitting the dataset.   Quick Change   Now as thats done  I want you to remember one thing from the last part where we set the d_model=vocab_size because we only had one layer which is Embedding.   Well  as now we have a proper mapping layers using Linear  we can change our embedding size to desired number and learn more representation of the character. Lets make it 512.   # used to define size of embeddings d_model = 512 # dont forget to add a linear layer which transforms embedding dim(d_model) to vocab_size class GPT(nn.Module): def __init__(self  vocab_size  d_model): ... self.linear1 = nn.Linear(d_model  vocab_size) def forward(self  inputs  targets = None): ... logits = self.ln2(logits + adn_logits) logits = self.linear1(logits) ... return logits  loss ...By doing just this change  we have completed full model of our GPT-2 Transformer decoder architecture:   But were not done yet. Lets continue improving the model..   c. (Masked) Multi-Head AttentionYou might already be familiar with the power of the Self-Attention mechanism and how it enhances a models ability to generalize contextual relationships within texts. But what if I told you theres a way for the model to understand different linguistic properties within the text  such as how words or characters are interconnected and their temporal usage? Imagine the model learning distinctions between consonants and vowels  and when and where to appropriately use them.   Sounds intriguing  doesnt it? While representing the overall sequence context in Self-Attention with d_model  we can now divide d_model into multiple heads. Each head will have its own sets of representations for Query  Key  and Value  enabling the model to learn multiple contextual nuances within the sequence. Let's enhance our attention layer by incorporating multiple heads.   Code:   n_heads = 4 # number of self-attention heads. should be divisible with d_model class MultiHeadAttention(nn.Module): def __init__(self  d_model: int  n_heads: int): super().__init__() self.n_heads = n_heads self.head_dim = d_model // n_heads assert (n_heads * self.head_dim == d_model) self.query = nn.Linear(d_model  d_model) self.key = nn.Linear(d_model  d_model) self.value = nn.Linear(d_model  d_model) self.fc_out = nn.Linear(d_model  d_model) self.dropout = nn.Dropout(0.2) def forward(self  inputs: torch.Tensor): B  seq_length  d_model = inputs.shape # Project the input embeddings into Q  K  and V Q = self.query(inputs).view(B  seq_length  self.num_heads  self.head_dim).permute(0  2  1  3) K = self.key(inputs).view(B  seq_length  self.num_heads  self.head_dim).permute(0  2  1  3) V = self.value(inputs).view(B  seq_length  self.num_heads  self.head_dim).permute(0  2  1  3) # Compute attention scores attention_scores = torch.matmul(Q  K.transpose(-2  -1)) / math.sqrt(self.head_dim) # Apply mask to prevent attention to future tokens mask = torch.triu(torch.ones(seq_length  seq_length)  diagonal=1).bool().to(inputs.device) attention_scores = attention_scores.masked_fill(mask  float('-inf')) attention_weights = torch.softmax(attention_scores  dim=-1) # Compute the weighted sum of the values attention_output = torch.matmul(self.dropout(attention_weights)  V) # Concatenate heads and put them back to the original shape attention_output = attention_output.permute(0  2  1  3).contiguous() attention_output = attention_output.view(B  seq_length  d_model) # Apply the final linear transformation out = self.fc_out(attention_output) return out class GPT(nn.Module): def __init__(self  vocab_size  d_model  n_heads): super().__init__() ... # replace selfattention layer with multiheadattention self.att = MultiHeadAttention(d_model  n_heads) ... m = GPT(vocab_size=vocab_size  d_model=d_model  n_heads=n_heads).to(device)Now  sit back  let the model train and see the magic   You should now see a significant improvement in the model performance and output. All thanks to the Multi-Head attention. You can play around with the head size to see if the model learn any better representations.   d. GPT Decoder BlocksIf you carefully go through the model diagrams presented throughout the project  you might notice I have starting adding few layers inside a rectangular blocks. They are called decoder blocks. And  just like we can add multiple layers of Linear network  we can also add multiple blocks of those group of models. Lets see how:   Well first take out our Attention  Layer Norms and Feed Forward network into a separate module GPTBlock.   class GPTBlock(nn.Module): def __init__(self  d_model  n_heads): super().__init__() self.att = MultiHeadAttention(d_model  n_heads) self.ln1 = nn.LayerNorm(d_model) self.ln2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(0.2) self.fcn = nn.Sequential( nn.Linear(d_model  4 * d_model)  nn.GELU()  nn.Linear(4 * d_model  d_model) ) def forward(self  logits): att_logits = self.att(logits) adn_logits = self.ln1(logits + att_logits) logits = self.dropout(adn_logits) logits = self.fcn(logits) logits = self.ln2(logits + adn_logits) return logitsNow modify our GPT class to incorporate the block in-place of all these layers inside it along with a constructor parameter n_layer to define number of decoder blocks/layers.   n_layers = 2 # number of gpt blocks/layers class GPT(nn.Module): def __init__(self  vocab_size  d_model  n_heads  n_layers): super().__init__() self.wte = nn.Embedding(vocab_size  d_model) # word token embeddings self.wpe = PositionalEncoding(context_length  d_model) # word position encodings self.blocks = nn.ModuleList([GPTBlock(d_model  n_heads) for _ in range(n_layers)]) self.linear1 = nn.Linear(d_model  vocab_size) def forward(self  inputs  targets = None): logits = self.wte(inputs) # dim -> batch_size  sequence_length  d_model logits = self.wpe(logits) for block in self.blocks: logits = block(logits) logits = self.linear1(logits) ... return logits  loss ... m = GPT(vocab_size=vocab_size  d_model=d_model  n_heads=n_heads  n_layers=n_layers).to(device)e. Improving TokenizerNow theres one more fix that I want to do in our code and its the Tokenizer. Yes  the character level tokenizer which has been overloading our model with tons of tokens with a very little information. Lets improve our tokenizer using the tiktoken library which is an official python library by OpenAI for GPT tokenizers.   The library uses Byte Pair Encoding(BPE) algorithm which creates merges of words or different section of words based on how often they appeared on training the tokenizer.   Installation:   pip install tiktokenCode:   import tiktoken tokenizer = tiktoken.get_encoding('gpt2') vocab_size = tokenizer.n_vocabWe have now increased our vocab size to 50257 which means model gets to see many variations of words and sequences.   Now lets encode our data using the new tokenizer. We will modify our data initialization as:   import torch # use cpu or gpu based on your system device = cpu if torch.cuda.is_available(): device = cuda data_dir = data.txt text = open(data_dir  'r').read() # load all the data as simple string # convert our text data into tokenized tensor data = torch.tensor(tokenizer.encode(text)  dtype=torch.long  device=device)Then  replace any calls to your previous encoding (encode) and decoding (decode) functions with tokenizer.encode() and tokenizer.decode()  respectively. This adjustment ensures compatibility with the new tokenizer.   f. Final GPT-2 TrainingU+1F973 We have finally reached towards the end of the project and quite a new learning experience. We just have to made few adjustments so that our model trains faster and better. And then we are good to go.   Lets do few changes. You can adjust these based on your requirements and system support.   context_length = 512 # number of tokens processed in a single batch d_model = 512 n_layers = 1 # number of gpt blocks/layers class GPT(nn.Module): def __init__(self  vocab_size  d_model  n_heads  n_layers): super().__init__() self.wte = nn.Embedding(vocab_size  d_model) # word token embeddings self.wpe = PositionalEncoding(context_length  d_model) # word position encodings self.blocks = nn.ModuleList([GPTBlock(d_model  n_heads) for _ in range(n_layers)]) self.linear1 = nn.Linear(d_model  vocab_size) # parameter sharing + self.wte.weight = self.linear1.weightTo learn more about parameter sharing in GPT-2  learn here.   You can visualize current model structure by just printing the model variable itself:   And just like that  we have built our own 29M GPT-2 Model which will be sufficient for our use case.   Now  before training our model  lets compile it using torch.compile. It ensures that almost all the matrix multiplications and other operations that happens within the model are mapped before hand. And in simple words  the model can directly compute the final stage by merging all the operations instead of going line by line or layer by layer.   m = GPT(vocab_size=vocab_size  d_model=d_model  n_heads=n_heads  n_layers=n_layers).to(device) m = torch.compile(m)Ive also modified our learning rate and training loop as below:   lr = 1e-3 optim = torch.optim.AdamW(m.parameters()  lr=lr  weight_decay=0.1) scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim  T_max=3000  eta_min=lr*0.1)epochs = 3500 eval_steps = 100 # perform evaluation in every n steps # store the losses train_loss = {} # train the model for e in range(epochs): xb  yb = train_loader.get_batch() logits  loss = m(xb  yb) optim.zero_grad(set_to_none=True) loss.backward() # gradient clipping torch.nn.utils.clip_grad_norm_(m.parameters()  max_norm=1) optim.step() scheduler.step() train_loss[e] = loss.item() if e % eval_steps == 0 or e == epochs-1: m.eval() with torch.no_grad(): xvb  yvb = eval_loader.get_batch() _  e_loss = m(xvb  yvb) print(fEpoch: {e+ep}\ttrain_loss: {loss:.4f}\teval_loss: {e_loss:.4f}) m.train() # Back to training modeAs the model has gotten much bigger now  I am using GoogleColab to train the model. You can acce this link to open on Colab.   After training for ~3500 epochs  I got the following training loss curve:   And finally  a song output from model U+1F3B6U+1F3B6:   There it was. A step-by-step guide to building a custom GPT-2 model and training on own data. Feel free to modify the hyper-parameters and layers to feed according to your data needs.   Thats it for this project. But I wont stop here. I have been planning for few new articles regarding improving model performance and training. So stay tuned. Happy learning :).