Important LLMs Papers for the Week from 08/07 to 14/07

Large language models (LLMs) have advanced rapidly in recent years. As new generations of models are developed  researchers and engineers need to stay informed on the latest progress.   This article summarizes some of the most important LLM papers published during the Second Week of July 2024. The papers cover various topics shaping the next generation of language models  from model optimization and scaling to reasoning  benchmarking  and enhancing performance.   Keeping up with novel LLM research across these domains will help guide continued progress toward models that are more capable  robust  and aligned with human values.   Table of Contents:LLM Progress & BenchmarkingLLM Training  Evaluation & InferenceLLM Fine-TuningLLM Quantization & AlignmentLLM ReasoningLLM Safety & AlignmentMost insights I share in Medium have previously been shared in my weekly newsletter  To Data & Beyond.   If you want to be up-to-date with the frenetic world of AI while also feeling inspired to take action or  at the very least  to be well-prepared for the future ahead of us  this is for you.   U+1F3DDSubscribe belowU+1F3DD to become an AI leader among your peers and receive content not present in any other platform  including Medium:   To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   1. LLM Progress & Benchmarking1.1. Learning to (Learn at Test Time): RNNs with Expressive Hidden StatesSelf-attention performs well in a long context but has quadratic complexity. Existing RNN layers have linear complexity  but their performance in a long context is limited by the expressive power of their hidden state. We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state.   The key idea is to make the hidden state a machine learning model itself  and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences  our layers are called Test-Time Training (TTT) layers.   We consider two instantiations: TTT-Linear and TTT-MLP  whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters  comparing with a strong Transformer and Mamba  a modern RNN.   Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer  they can keep reducing perplexity by conditioning on more tokens  while Mamba cannot after 16k context.   With preliminary systems optimization  TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time. TTT-MLP still faces challenges in memory I/O  but shows larger potential in long context  pointing to a promising direction for future research.   View arXiv pageView PDF1.2. LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 LanguagesLarge Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks  yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training.   To address this  we dedicate 35 000 A100-SXM480GB GPU hours to conducting extensive multilingual continual pre-training on the LLaMA series models  enabling translation support across more than 100 languages.   Through a comprehensive analysis of training strategies  such as vocabulary expansion and data augmentation  we develop LLaMAX. Remarkably  without sacrificing its generalization ability  LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs~(by more than 10 spBLEU points) and performs on-par with specialized translation model~(M2M-10012B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model.   Project pageView arXiv pageView PDF1.3. GTA: A Benchmark for General Tool AgentsSignificant focus has been placed on integrating large language models (LLMs) with various tools in developing general-purpose agents. This poses a challenge to LLMs tool-use capabilities. However  there are evident gaps between existing tool-use evaluations and real-world scenarios.   Current evaluations often use AI-generated queries  single-step tasks  dummy tools  and text-only interactions  failing to reveal the agents real-world problem-solving abilities effectively.   To address this  we propose GTA  a benchmark for General Tool Agents  featuring three main aspects:   Real user queries: human-written queries with simple real-world objectives but implicit tool-use  requiring the LLM to reason the suitable tools and plan the solution steps.Real deployed tools: an evaluation platform equipped with tools across perception  operation  logic  and creativity categories to evaluate the agents actual task execution performance.Real multimodal inputs: authentic image files  such as spatial scenes  web page screenshots  tables  code snippets  and printed/handwritten materials  used as the query contexts to align with real-world scenarios closely. We design 229 real-world tasks and executable tool chains to evaluate mainstream LLMs.Our findings show that real-world user queries are challenging for existing LLMs  with GPT-4 completing less than 50% of the tasks and most LLMs achieving below 25%.   This evaluation reveals the bottlenecks in the tool-use capabilities of current LLMs in real-world scenarios  which provides future direction for advancing general-purpose tool agents.   Project pageView arXiv pageView PDF1.4. TheoremLlama: Transforming General-Purpose LLMs into Lean4 ExpertsProving mathematical theorems using computer-verifiable formal languages like Lean significantly impacts mathematical reasoning. One approach to formal theorem proving involves generating complete proofs using Large Language Models (LLMs) based on Natural Language (NL) proofs.   Similar methods have shown promising results in code generation. However  most modern LLMs exhibit suboptimal performance due to the scarcity of aligned NL and Formal Language (FL) theorem-proving data.   This scarcity results in a paucity of methodologies for training LLMs and techniques to fully utilize their capabilities in composing formal proofs. To address the challenges  this paper proposes TheoremLlama  an end-to-end framework to train a general-purpose LLM to become a Lean4 expert.   This framework encompasses NL-FL aligned dataset generation methods  training approaches for the LLM formal theorem prover  and techniques for LLM Lean4 proof writing.   Using the dataset generation method  we provide Open Bootstrapped Theorems (OBT)  an NL-FL aligned and bootstrapped dataset. A key innovation in this framework is the NL-FL bootstrapping method  where NL proofs are integrated into Lean4 code for training datasets  leveraging the NL reasoning ability of LLMs for formal reasoning.   The TheoremLlama framework achieves cumulative accuracies of 36.48% and 33.61% on MiniF2F-Valid and Test datasets respectively  surpassing the GPT-4 baseline of 22.95% and 25.41%. We have also open-sourced our model checkpoints and generated dataset  and will soon make all the code publicly available.   View arXiv pageView PDF1.5. SEED-Story: Multimodal Long Story Generation with Large Language ModelWith the remarkable advancements in image generation and open-form text generation  the creation of interleaved image-text content has become an increasingly intriguing field.   Multimodal story generation  characterized by producing narrative texts and vivid images in an interleaved manner  has emerged as a valuable and practical task with broad applications.   However  this task poses significant challenges  as it necessitates the comprehension of the complex interplay between texts and images  and the ability to generate long sequences of coherent  contextually relevant texts and visuals. In this work  we propose SEED-Story  a novel method that leverages a Multimodal Large Language Model (MLLM) to generate extended multimodal stories.   Our model  built upon the powerful comprehension capability of MLLM  predicts text tokens as well as visual tokens  which are subsequently processed with an adapted visual de-tokenizer to produce images with consistent characters and styles.   We further propose a multimodal attention sink mechanism to enable the generation of stories with up to 25 sequences (only 10 for training) in a highly efficient autoregressive manner.   Additionally  we present a large-scale and high-resolution dataset named StoryStream for training our model and quantitatively evaluating the task of multimodal story generation in various aspects.   View arXiv pageView PDF1.6. Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense KnowledgeHumans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However  existing works focus on   Image-sharing behavior in singular sessions leads to limited long-term social interactionA lack of personalized image-sharing behavior.In this work  we introduce Stark  a large-scale long-term multi-modal conversation dataset that covers a wide range of social personas in a multi-modality format  time intervals  and images.   To construct Stark automatically  we propose a novel multi-modal contextualization framework  Mcu  that generates long-term multi-modal dialogue distilled from ChatGPT and our proposed Plan-and-Execute image aligner.   Using our Stark  we train a multi-modal conversation model  Ultron 7B  which demonstrates impressive visual imagination ability. Furthermore  we demonstrate the effectiveness of our dataset in human evaluation. We make our source code and dataset publicly available.   View arXiv pageView PDFAdd to collection   2. LLM Training  Evaluation & Inference2.1. HEMM: Holistic Evaluation of Multimodal Foundation ModelsMultimodal foundation models that can holistically process text alongside images  video  audio  and other sensory modalities are increasingly used in a variety of real-world applications.   However  it is challenging to characterize and study progress in multimodal foundation models  given the range of possible modeling decisions  tasks  and domains.   In this paper  we introduce a Holistic Evaluation of Multimodal Models (HEMM) to systematically evaluate the capabilities of multimodal foundation models across a set of 3 dimensions: basic skills  information flow  and real-world use cases.   Basic multimodal skills are internal abilities required to solve problems  such as learning interactions across modalities  fine-grained alignment  multi-step reasoning  and the ability to handle external knowledge. Information flow studies how multimodal content changes during a task through querying  translation  editing  and fusion.   Use cases span domain-specific challenges introduced in real-world multimedia  affective computing  natural sciences  healthcare  and human-computer interaction applications.   Through comprehensive experiments across the 30 tasks in HEMM  they   Identify key dataset dimensions (e.g.  basic skills  information flows  and use cases) that pose challenges to todays modelsDistill performance trends regarding how different modeling dimensions (e.g.  scale  pre-training data  multimodal alignment  pre-training  and instruction tuning objectives) influence performance.The conclusions regarding challenging multimodal interactions use cases  and tasks requiring reasoning and external knowledge  the benefits of data and model scale  and the impacts of instruction tuning yield actionable insights for future work in multimodal foundation models.   View arXiv pageView PDF2.2. On Leakage of Code Generation Evaluation DatasetsIn this paper  we consider contamination by code generation test sets  in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them:   Direct data leakage Indirect data leakage through the use of synthetic dataOverfitting to evaluation sets during model selection.Project pageView arXiv pageView PDF3. LLM Fine-Tuning3.1. InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with Inverse-InstructRecent advancements in open-source code large language models (LLMs) have demonstrated remarkable coding abilities by fine-tuning the data generated from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction tuning.   This paper explores how to further improve an instruction-tuned code LLM by generating data from itself rather than querying closed-source LLMs. Our key observation is the misalignment between the translation of formal and informal languages: translating formal language (i.e.  code) to informal language (i.e.  natural language) is more straightforward than the reverse.   Based on this observation  we propose INVERSE-INSTRUCT  which summarizes instructions from code snippets instead of the reverse. Specifically  given an instruction-tuning corpus for code and the resulting instruction-tuned code LLM  we ask the code LLM to generate additional high-quality instructions for the original corpus through code summarization and self-evaluation.   Then  we fine-tune the base LLM on the combination of the original corpus and the self-generated one  which yields a stronger instruction-tuned LLM. We present a series of code LLMs named InverseCoder  which surpasses the performance of the original code LLMs on a wide range of benchmarks  including Python text-to-code generation  multilingual coding  and data-science code generation.   View arXiv pageView PDF3.2. AgentInstruct: Toward Generative Teaching with Agentic FlowsSynthetic data is becoming increasingly important for accelerating the development of language models  both large and small. Despite several successful use cases  researchers also raised concerns about model collapse and the drawbacks of imitating other models.   This discrepancy can be attributed to the fact that synthetic data varies in quality and diversity. Effective use of synthetic data usually requires significant human effort in curating the data.   We focus on using synthetic data for post-training  specifically creating data by powerful models to teach a new skill or behavior to another model  we refer to this setting as Generative Teaching.   We introduce AgentInstruct  an extensible agentic framework for automatically creating large amounts of diverse and high-quality synthetic data. AgentInstruct can create both the prompts and responses  using only raw data sources like text documents and code files as seeds.   We demonstrate the utility of AgentInstruct by creating a post-training dataset of 25M pairs to teach language models different skills  such as text editing  creative writing  tool usage  coding  reading comprehension  etc.   The dataset can be used for instruction tuning of any base model. We post-train Mistral-7b with the data. When comparing the resulting model Orca-3 to Mistral-7b-Instruct (which uses the same base model)  we observe significant improvements across many benchmarks.   For example  40% improvement on AGIEval  19% improvement on MMLU  54% improvement on GSM8K  38% improvement on BBH  and 45% improvement on AlpacaEval. Additionally  it consistently outperforms other models such as LLAMA-8B-instruct and GPT-3.5-turbo.   View arXiv pageView PDF4. LLM Quantization4.1. Inference Performance Optimization for Large Language Models on CPUsLarge language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. However  the deployment of LLMs with high performance in low-resource environments has garnered significant attention in the industry.   When GPU hardware resources are limited  we can explore alternative options on CPUs. To mitigate the financial burden and alleviate constraints imposed by hardware resources  optimizing inference performance is necessary. In this paper  we introduce an easily deployable inference performance optimization solution aimed at accelerating LLMs on CPUs.   In this solution  we implement an effective way to reduce the KV cache size while ensuring precision. We propose a distributed inference optimization approach and implement it based on oneAPI Collective Communications Library.   Furthermore  we propose optimization approaches for LLMs on CPU and conduct tailored optimizations for the most commonly used models.   Project pageView arXiv pageView PDF5. LLM Reasoning5.1. ChartGemma: Visual Instruction-tuning for Chart Reasoning in the WildGiven the ubiquity of charts as a data analysis  visualization  and decision-making tool across industries and sciences  there has been a growing interest in developing pre-trained foundation models as well as general-purpose instruction-tuned models for chart understanding and reasoning.   However  existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts  ignoring the visual trends and patterns in chart images  and use weakly aligned vision-language backbone models for domain-specific training  limiting their generalizability when encountering charts in the wild.   We address these important drawbacks and introduce ChartGemma  a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables  ChartGemma is trained on instruction-tuning data generated directly from chart images  thus capturing both high-level trends and low-level visual information from a diverse set of charts.   Our simple approach achieves state-of-the-art results across 5 benchmarks spanning chart summarization  question answering  and fact-checking  and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries.   Project pageView arXiv pageView PDF5.2. Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with ChecklistExceptional mathematical reasoning ability is one of the key features that demonstrate the power of large language models (LLMs). How to comprehensively define and evaluate the mathematical abilities of LLMs  and even reflect the user experience in real-world scenarios  has emerged as a critical issue.   Current benchmarks predominantly concentrate on problem-solving capabilities  which presents a substantial risk of model overfitting and fails to accurately represent genuine mathematical reasoning abilities. In this paper  we argue that if a model really understands a problem  it should be robustly and readily applied across a diverse array of tasks.   Motivated by this  we introduce MATHCHECK  a well-designed checklist for testing task generalization and reasoning robustness  as well as an automatic tool to generate checklists efficiently.   MATHCHECK includes multiple mathematical reasoning tasks and robustness test types to facilitate a comprehensive evaluation of both mathematical reasoning ability and behavior testing.   Utilizing MATHCHECK  we develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual reasoning and multi-modal reasoning capabilities  respectively  serving as upgraded versions of benchmarks including GSM8k  GeoQA  UniGeo  and Geometry3K.   We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs  assessing their comprehensive mathematical reasoning abilities.   Our results demonstrate that while frontier LLMs like GPT-4o continue to excel in various abilities on the checklist  many other model families exhibit a significant decline.   Further experiments indicate that  compared to traditional math benchmarks  MATHCHECK better reflects true mathematical abilities and represents mathematical intelligence more linearly  thereby supporting our design. On our MATHCHECK  we can easily conduct detailed behavior analysis to deeply investigate models.   View arXiv pageView PDF5.3. Self-Recognition in Language ModelsA rapidly growing number of applications rely on a small set of closed-source language models (LMs). This dependency might introduce novel security risks if LMs develop self-recognition capabilities. Inspired by human identity verification methods  we propose a novel approach for assessing self-recognition in LMs using model-generated security questions.   Our test can be externally administered to keep track of frontier models as it does not require access to internal model parameters or output probabilities. We use our test to examine self-recognition in ten of the most capable open- and closed-source LMs currently publicly available.   Our extensive experiments found no empirical evidence of general or consistent self-recognition in any examined LM. Instead  our results suggest that given a set of alternatives  LMs seek to pick the best answer  regardless of its origin.   Moreover  we find indications that preferences about which models produce the best answers are consistent across LMs. We additionally uncover novel insights on position bias considerations for LMs in multiple-choice settings.   View arXiv pageView PDF5.4. Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models  The Story Goes OnIn this paper  we investigate the underlying factors that potentially enhance the mathematical reasoning capabilities of large language models (LLMs). We argue that the data scaling law for math reasoning capabilities in modern LLMs is far from being saturated  highlighting how the models quality improves with increases in data quantity.   To support this claim  we introduce the Skywork-Math model series  supervised fine-tuned (SFT) on common 7B LLMs using our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved impressive accuracies of 51.2% on the competition-level MATH benchmark and 83.9% on the GSM8K benchmark using only SFT data  outperforming an early version of GPT-4 on MATH.   The superior performance of Skywork-Math models contributes to our novel two-stage data synthesis and model SFT pipelines  which include three different augmentation methods and a diverse seed problem set  ensuring both the quantity and quality of the Skywork-MathQA dataset across varying difficulty levels.   Most importantly  we provide several practical takeaways to enhance math reasoning abilities in LLMs for both research and industry applications.   View arXiv pageView PDF6. LLM Safety & Alignment6.1. Safe Unlearning: A Surprisingly Effective and Generalizable Solution to Defend Against Jailbreak AttacksLLMs are known to be vulnerable to jailbreak attacks  even after safety alignment. An important observation is that  while different types of jailbreak attacks can generate significantly different queries  they mostly result in similar responses that are rooted in the same harmful knowledge (e.g.  detailed steps to make a bomb).   Therefore  we conjecture that directly unlearning the harmful knowledge in the LLM can be a more effective way to defend against jailbreak attacks than the mainstream supervised fine-tuning (SFT) based approaches.   Our extensive experiments confirmed our insight and suggested the surprising generalizability of our unlearning-based approach: using only 20 raw harmful questions without any jailbreak prompt during training  our solution reduced the Attack Success Rate (ASR) in Vicuna-7B on out-of-distribution (OOD) harmful questions wrapped with various complex jailbreak prompts from 82.6\% to 7.7\%.   This significantly outperforms Llama27B-Chat  which is fine-tuned on about 0.1M safety alignment samples but still has an ASR of 21.9\% even with the help of an additional safety system prompt.   Further analysis reveals that the generalization ability of our solution stems from the intrinsic relatedness among harmful responses across harmful questions (e.g.  response patterns  shared steps and actions  and similarity among their learned representations in the LLM).   Project PageView arXiv pageView PDF7. Transformers & Attention Models7.1. Associative Recurrent Memory TransformerThis paper addresses the challenge of creating a neural architecture for very long sequences that require constant time for processing new information at each time step.   Our approach  Associative Recurrent Memory Transformer (ARMT)  is based on transformer self-attention for local context and segment-level recurrence for storage of task-specific information distributed over a long context.   We demonstrate that ARMT outperforms existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%.   View arXiv pageView PDF8. LLM Agents8.1. AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM AgentsAdvancements in generative AI have broadened the potential applications of Large Language Models (LLMs) in the development of autonomous agents. Achieving true autonomy requires accumulating and updating knowledge gained from interactions with the environment and effectively utilizing it.   Current LLM-based approaches leverage past experiences using a full history of observations  summarization  or retrieval augmentation. However  these unstructured memory representations do not facilitate the reasoning and planning essential for complex decision-making.   In our study  we introduce AriGraph  a novel method wherein the agent constructs a memory graph that integrates semantic and episodic memories while exploring the environment.   This graph structure facilitates efficient associative retrieval of interconnected concepts  relevant to the agents current state and goals  thus serving as an effective environmental model that enhances the agents exploratory and planning capabilities.   We demonstrate that our Ariadne LLM agent  equipped with this proposed memory architecture augmented with planning and decision-making  effectively handles complex tasks on a zero-shot basis in the TextWorld environment.   Our approach markedly outperforms established methods such as full-history  summarization  and Retrieval-Augmented Generation in various tasks  including the cooking challenge from the First TextWorld Problems competition and novel tasks like house cleaning and puzzle Treasure Hunting.   View arXiv pageView PDFIf you like the article and would like to support me  make sure to:U+1F44F Clap for the story (50 claps) to help this article be featuredSubscribe to To Data & Beyond NewsletterFollow me on MediumU+1F4F0 View more content on my medium profileU+1F514 Follow Me: LinkedIn U+007CYoutube U+007C GitHub U+007C TwitterSubscribe to my newsletter To Data & Beyond to get full and early access to my articles:To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   Are you looking to start a career in data science and AI and do not know how? I offer data science mentoring sessions and long-term career mentoring:Mentoring sessions: https://lnkd.in/dXeg3KPWLong-term mentoring: https://lnkd.in/dtdUYBrM


Bayesian analysis and decision theory: application to determine a decision point for classification problems

A dilemma often presented in classification problems where the output is a number is determining the cutout point between the categories. For example  the output of a neural network might be a number between 0 and 1  lets say 0.7  does that correspond to the positive (1) category or to the negative (0) category? Common sense says to use 0.5 as a decision marker  but what if there is a higher risk in underestimating the positives? or if the classes are unbalanced?   A correct estimation of the cut point in these cases warrants some review of probabilities and Bayesian theory. When talking about probabilities  three rules take the central stage for the processes that will follow:   Sum rule:Where considering x and y as two events  the probability of x is the sum of the x occurring together with each option of y.   Product rule:This means that the probability of x and y occurring together is equal to the probability of y occurring given that x happened time the probability of x occurring.   Bayes theorem:Bayes theorem is a very powerful tool that provides a way to update the probabilities of an event (in this case  event y) after getting some new information  represented in this case by p(xU+007Cy). The new  updated probability is then p(yU+007Cx).   In detail  p(y) is named the prior  the probability of y before the new information is obtained; p(xU+007Cy) is the probability of a new event x happening provided that y exists  this is the new data or information about the system; and p(x) is the marginal probability of the event x regardless of the value of y.   Bayes theorem can be expressed in any of the following forms  which all are derived from the original equation and the two rules explained above:   To illustrate the power of Bayes theorem I will use an example. Lets say that having a disease is event Y (not having it would be Y0  and Y1 is the unfortunate event of being sick); and getting a positive blood test to detect the disease is the event X. The probability of having the disease  over the whole population is a small number p(y). About the test  someone that has the disease will test positive with a probability of p(xU+007Cy); and the percentage of the population that will test positive  regardless if they are sick or not is p(x)  which includes then the real positives and the false positives.   Lets plug some numbers for illustration:   p(y) = Prob. of having the disease  or people sick over the whole population: 1 in 10 000 = 0.0001   p(xU+007Cy) = probability of getting a positive test if there is a disease (the effectivity of the test itself): 0.9 / the test is effective in locating the disease 90% of the time   p(x) = probability of positive test / it is the number of people that get the test and test positive  regardless of whether they being really sick or not: 1 in 1000   With this  applying Bayes theorem: p(yU+007Cx) = (0.9*0.0001)/(0.001) = 9%   This means that even after testing positive  the actual chances of having the disease are still low  and more tests are needed to produce a diagnosis.   After applying Bayes theorem  the probability of having the disease for this individual has been updated from 1 in 10 000 to almost 1 in 10.   In reality  these blood tests  just as the numerical outcome of both regression and classification problems in neural networks  are not binary but formed by a continuous variable. In this situation  the question is where to cut the results and assign a positive or negative value to the outcome. Common sense dictates to use the middle point (0.5 if the last layer is a softmax for example)  but that is not the only option and ignores issues like different risks or unbalanced training variables.   Considering the risks is very important in the example used above  because getting a false positive (test positive but not being really sick) only carries the small risk of being annoyed by further testing  but a false negative (being sick and getting a negative test) means further spread of the disease and failure to receive care for it.   The next chart shows what the distributions look like  the blue one being the healthy individuals distribution and the red one the sick ones. The X axis is the test result (for example a value of protein xxx in the blood)  and the Y axis is a value representing quantity. As these are probability distributions  they are normalized so that the area under them totals to one.   import numpy as np import matplotlib.pyplot as plt import scipy #define mean and standard dev mu  sg = 10  1 #serie of 100000 points s = np.random.normal(mu  sigma  100000) #plot the histogram and create bins count  bins  ignored = plt.hist(s  500  density=True) #standard distribution formula def standardDistribution(mu sg x): y = (1/np.sqrt(2*np.pi*sg**2))*np.exp(-((x-mu)**2)/(2*sg**2)) return y #prob distribution of negative test and values of test (x) #for negative test mu0  sg0 = 50  15 x = np.arange(0.0  150.0  0.01) probY0_X = standardDistribution(mu0 sg0 x) #for positive test mu1  sg1 = 100  20 x = np.arange(0.0  150.0  0.01) probY1_X = standardDistribution(mu1 sg1 x) fig  (ax1  ax2) = plt.subplots(1  2 sharex=True  sharey=True  figsize=(15 5)) ax1.plot(x  probY0_X  linewidth=2  color='b') ax1.plot(x  probY1_X  linewidth=2  color='r') ax1.set_title('The joined Y0 and Y1 with X') ax2.plot(x  probY1_X+probY0_X  linewidth=2  color='g') ax2.set_title('Probability of X')If we dont know anything about the individuals  if they are sick or not  we will only see the green chart  which is the distribution probability of the results of the test. We can see by intuition that there are two modes  which correspond to the median of the sick or healthy cases.   Note that in this process  I am going to assume that both distributions are normal or close to normal  which will be the case if the average of a significant number of random samples (central limit theorem).   Lets review in detail the first chart  we see four regions that are of interest in our case:   True positive: TP -> Good! accurate identification of the classTrue negative: TN -> Good! accurate identification of the classFalse negative: FN -> Bad! The result is attributed to class 0 (no disease in our example) when it really is class 1False positive: FP -> Bad! The result is attributed to class 1 when it belongs to class 0The areas of 3 and 4 measure how wrong the results are so  this is a good error function to minimize in order to get the best results of the model:   The last equation just requires remembering that these joint probabilities are Gaussian.   For more than two outcomes  the error area is generalized to:   At this point is easy to introduce bias to the error to account for risk. In our example  for the bad results  we want to penalize the false negative. We introduce to the error calculation factors Rfn and Rfp to account for their respective penalties.   At this point we have an optimization problem to find the minimum of the function of the error area.   The derivatives of the integrals are Gaussians   M is the cutting point that minimizes the error as we have defined it  given the assigned risk to each error type.   The next step is to resolve this last equation  what I am going to do in Python:   #formula to solve #for negative test mu0  sg0 = 50  15 #for positive test mu1  sg1 = 100  20 def func(w): r = (rFN/sg1)*(np.exp(-((w-mu1)**2)/(2*sg1**2))) - (rFP/sg0)*(np.exp(-((w-mu0)**2)/(2*sg0**2))) return r #sol no penalty rFN  rFP = 1  1 sol0 = scipy.optimize.fsolve(func x0=60) #sol penalty 5:1 rFN  rFP = 5  1 sol1 = scipy.optimize.fsolve(func x0=60) #sol penalty 10:1 rFN  rFP = 10  1 sol2 = scipy.optimize.fsolve(func x0=60) #plot with the solutions plt.figure(figsize=(12  10)) plt.plot(x  probY0_X  linewidth=1  color='b'  label='Y0 -> healthy') plt.plot(x  probY1_X  linewidth=1  color='r'  label='Y1 -> disease') plt.axvline(x=sol0  color='black'  ls='--'  label='Cut  no penalty') plt.axvline(x=sol1  color='gray'  ls='--'  label='Cut  penalty 1:5') plt.axvline(x=sol2  color='brown'  ls='--'  label='Cut  penalty 1:10') plt.legend(bbox_to_anchor=(1.0  1)  loc='upper left') plt.show()The vertical lines represent different solutions for the best point M with different weights or penalties; illustrating the impact of the manually introduced difference between the categories.   Applying Bayes theorem  these are the same results over the posterior functions p(YU+007CX):   #plot of p(YU+007Cx) for Y0 and Y1 plt.figure(figsize=(12  10)) plt.plot(x  probY0_X/(probY1_X + probY0_X)  linewidth=1  color='b'  label='Y0 -> healthy') plt.plot(x  probY1_X/(probY1_X + probY0_X)  linewidth=1  color='r'  label='Y1 -> disease') plt.axvline(x=sol0  color='black'  ls='--'  label='Cut  no penalty') plt.axvline(x=sol1  color='gray'  ls='--'  label='Cut  penalty 1:5') plt.axvline(x=sol2  color='brown'  ls='--'  label='Cut  penalty 1:10') plt.legend(bbox_to_anchor=(1.0  1)  loc='upper left') plt.show()In a real-life scenario  for machine learning  we can attack a problem of this same kind of optimization in three different ways:   Use the p(y x)  the probability of y and x occurring  as I just did above (which are the two distributions of having a blood value x and having the disease and not having the disease) for the training set. Then  determine the best point to cut.Use the posterior p(YU+007CX); which are probabilities of having the disease given a test result as data. The cut point is also determined as an optimization problem.Train a direct classification model with binary output  in the training set make sure the labels account for the different risk or resample in case of unbalanced classes. This method can be quicker  but it has several drawbacks  for example  it does not give much information about possible factors (problems in real life are generally multivariable)  removes the possibility of manual accounting for risk and has no option to reject low confidence results (close to the decision point).


A Complete Guide to Descriptive Statistics  Central Tendency and Dispersion

In a world filled with data  statistics is the compass guiding us through the huge seas of numbers. Statistics play an important role in predicting the weather  analyzing market trends  or assessing public health.   In this blog  well understand the essence of statistics  diving into one of its main branches: descriptive statistics.   But before starting with descriptive statistics  lets take a step back and understand what exactly statistics is. And why is it so crucial?   What is Statistics?According to Wikipedia:   Statistics is the discipline that concerns the collection  organization  analysis  interpretation  and presentation of data.   In simple terms  statistics means collecting information  summarizing  and determining what it means. Statistics helps us understand the patterns and trends within the data.   A world without Statistics?   Without statistics  we will never be able to understand how data behaves  what happened  or what may happen in the future.   All these things require a fundamental understanding of statistics.   With that context  lets get started with the topic for today i.e.  descriptive statistics.   Descriptive Statistics: Painting a Picture of DataDescriptive statistics help us summarize and describe the main features of a data set.   Imagine you have a dataset of students test scores. Descriptive statistics will tell you the average score  the range of scores  and how scores are distributed. It provides a snapshot and a concise overview of the data at hand.   Key Concepts in Descriptive Statistics1. Measures of Central TendencyA single number/statistic that quantifies the central behavior of the dataset. The central tendency can be measured using the following statistics:   i) Mean: Mean or arithmetic mean is the average of a set of numbers. For a dataset with n values  the mean is calculated using the following formula:   Mean uses all data points  providing a comprehensive measure of the numerical columns.   ii) Median: The middle value of a set of numbers arranged in ascending order.   For an even set of numbers  the median is the average of the middle 2 numbers while for the odd set of numbers  its the middle number.Just like the mean  the median can be applied to numeric data only.Median does not use all data points  potentially losing some information.If there are outliers in a numerical column  the preferred way to measure central tendency is the median (as outliers influence mean value but not median).iii) Mode: The most frequently occurring score. A dataset can have one mode (unimodal)  more than one mode (bimodal or multimodal)  or no mode at all if no number repeats.   How to Find the Mode:   Identify the Frequency: Count how many times each value appears in the dataset.Determine the Most Frequent Value: The value with the highest frequency is the mode.When to use Mode:   When analyzing categorical data where you want to know the most common category.When dealing with discrete data and interested in the most frequent value.When examining data distributions that are not symmetrical.But Central Tendency is not sufficient?   While Central Tendency measures provide important information about where the data is centered  they do not provide a complete picture of the datas distribution. Relying solely on measures of central tendency can be misleading  as they do not capture the variability or spread of the data.   Datasets with the same mean  median  or mode can have very different distributions. For example:   Example 1: Consider two datasets with the same mean:   Dataset A: [50  50  50  50  50]Dataset B: [10  30  50  70  90]Both have a mean of 50  but Dataset A has no variability  while Dataset B has a wide range of values. The mean alone does not capture this difference.   2. Measures of DispersionMeasures of dispersion quantify the spread or variability of the data points around the central value.   i) Range: The range is the difference between the maximum and minimum values in a column. It tells us the span of the data and gives a basic indication of the spread.   How to Calculate the Range:   Identify the maximum value in the dataset.Identify the minimum value in the dataset.Subtract the minimum value from the maximum value.Range=Maximum ValueMinimum Value   When to Use the Range:   When you need a quick and simple measure of dispersion.When comparing the variability of two or more datasets.In preliminary data analysis to get an overview of the data spread.ii) Variance: Variance measures the average squared deviation of each data point from the mean. In simpler terms  it tells us how spread out the data points are around the mean.   A higher variance indicates that the data points are more spread out  while a lower variance indicates that they are closer to the mean.   The formula for Variance: The formula for variance differs slightly depending on whether we are dealing with a population or a sample.   When to use:   Variance provides a good understanding of the distribution of values within the dataset (around the mean).iii) Standard Deviation: The square root of the variance  indicating how spread out the scores are around the mean.   Formula for Standard Deviation: Just like Variance  the formula for standard deviation depends on whether you are dealing with a population or a sample.   Variance vs Standard Deviation  Which one to use when?Mathematical Properties and Interpretation:   Variance: Provides a measure that is useful for mathematical and statistical calculations. It is often used in theoretical contexts where the squaring of deviations simplifies further mathematical manipulation.Standard Deviation: Offers a more intuitive measure of spread  as it is in the same units as the data  making it easier to interpret.Analytical Convenience(more on this in future blogs):   Variance: In many statistical formulas and tests (e.g.  ANOVA  regression analysis)  working with variance is more convenient because of its additive properties.Standard Deviation: When communicating results to a non-technical audience or comparing the spread of different datasets  standard deviation is preferred due to its direct interpretability.SummaryStatistics is a powerful tool that helps us make sense of data  uncover patterns and making informed decisions. Descriptive statistics provide a summary of the data  giving us insights into its central tendencies and variability.   So next time you come across a data set  remember to use the power of statistics in order to turn those numbers into meaningful insights.


10 Important Blogs to Stay Updated with LLM Research & News

Staying up-to-date with the rapidly evolving world of Large Language Model (LLM) research and news can be a challenging task. With countless resources and endless streams of information  its easy to get overwhelmed. Luckily  there are many outstanding bloggers and newsletter writers who dedicate their time to distilling the latest advancements and trends in LLM research.   This blog post aims to be a comprehensive guide  curating ten of the most informative and insightful blogs and newsletters for anyone interested in staying informed about LLMs. From established researchers and engineers to passionate individuals sharing their insights  these sources cover various aspects of LLM development  applications  and ethical considerations.   Whether youre a seasoned LLM researcher or a novice enthusiast  the resources highlighted in this blog will provide you with in-depth analyses  insightful commentary  and a front-row seat to the exciting world of LLMs. Each offers a unique perspective  helping readers navigate the complex landscape of this fascinating field.   From there  readers can explore each bloggers work  gaining a deeper understanding of the current state and future of LLMs and the impact they have on various industries and society at large.   To Data & Beyond Newsletter by Youssef HosniAhead of AI Newsletter by Sebastian RaschkaChip Huyen BlogEugene Yan BlogPhilipp Schmid BlogJason Liu BlogHamel Husain BlogSimon Willison BlogOmar Sanseviero BlogLilian Weng BlogMost insights I share in Medium have previously been shared in my weekly newsletter  To Data & Beyond.   If you want to be up-to-date with the frenetic world of AI while also feeling inspired to take action or  at the very least  to be well-prepared for the future ahead of us  this is for you.   U+1F3DDSubscribe belowU+1F3DD to become an AI leader among your peers and receive content not present in any other platform  including Medium:   To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   1. To Data & Beyond Newsletter By Youssef HosniThe To Data & Beyond newsletter by Youssef Hosni is an excellent resource for staying updated with the latest research and developments in large language models (LLMs).   It offers in-depth analysis  summaries of recent research papers  and discussions on trends in data science and machine learning. The newsletter aims to provide valuable insights for both professionals and enthusiasts in the field  making complex topics more accessible.   2. Ahead of AI Newsletter by Sebastian RaschkaAhead of AI newsletter authored by Sebastian Raschka  is a highly regarded newsletter that provides in-depth coverage of the latest research and developments in AI  particularly focusing on machine learning and large language models (LLMs). Also  he focuses in his newsletter on fine-tuning LLMs with different techniques.   With over a decade of experience in AI and a passion for education  Raschka curates content that is valuable for both researchers and practitioners aiming to stay ahead in the rapidly evolving AI field.   3. Chip Huyen BlogChip Huyens blog is an excellent resource for staying updated with recent research and developments in large language models (LLMs) and AI. Her posts often delve deeply into technical concepts  providing in-depth analysis and insights.   For those interested in following her updates more closely  she aims to post once a month and hosts discussions on her Discord server. Subscribing to her newsletter is also a good way to stay informed about her latest posts and insights.   4. Eugene Yan BlogEugene Yans blog is a rich resource for staying updated on machine learning  data science  and large language models (LLMs). His blog features a variety of topics  including technical tutorials  system designs  practical tips for ML projects  and his personal experiences in the field.   Eugenes blog also includes summaries and reviews of industry practices  reflections on personal and professional growth  and strategies for leading data science teams effectively. This blend of technical depth and practical advice makes his blog a valuable resource for anyone involved in data science and machine learning.   5. Philipp Schmid BlogPhilipp Schmids blog is a valuable resource for anyone interested in staying updated with large language model (LLM) research and advancements.   Philipp Schmid provides detailed tutorials on cutting-edge topics like fine-tuning large language models  using reinforcement learning from human feedback (RLHF)  and optimizing models with DeepSpeed. His posts often include code snippets  configurations  and step-by-step instructions  making complex concepts accessible and actionable.   He also shares insights on optimizing model performance and efficiency  such as using mixed precision training and CPU offloading. These tips are crucial for practitioners who need to balance computational resources and model accuracy.   6. Jason Liu BlogJason Lius blog is a valuable resource for those interested in machine learning and large language models  offering detailed summaries of his research  deep dives into technical methodologies  and practical problem-solving examples. His writings are a mix of consulting  open source  personal work  and applying llms.   7. Hamel Husain BlogHamel Husains blog is an excellent resource for staying updated with the latest research and developments in Large Language Models (LLMs) and AI.   As a seasoned machine learning engineer with extensive experience at companies like Airbnb and GitHub  Hamel offers valuable insights into practical AI applications.   His blog covers a range of topics  including the operationalization of LLMs  debugging AI with adversarial validation  the utility of fine-tuning models  and optimizing LLM latency.   For instance  his post Is Fine-Tuning Still Valuable? delves into scenarios where fine-tuning significantly enhances performance  which is particularly insightful for practitioners debating the merits of this technique. Additionally  posts like vLLM & Large Models provide technical guidance on deploying large models using tensor parallelism across multiple GPUs.   Regularly updated and rich with technical details and real-world examples  Hamels blog is a must-read for AI researchers and practitioners aiming to keep abreast of cutting-edge LLM advancements   8. Simon Willison BlogSimon Willisons blog is a valuable resource for staying updated on the latest developments in large language models (LLMs) and machine learning.   Willison  a seasoned software engineer and co-creator of the Django web framework  offers in-depth insights into various aspects of LLMs  including their applications  ethical considerations  and technological advancements.   His posts cover a wide range of topics  such as the feasibility of running LLMs on personal devices  the impact of open-source models like Stanfords Alpaca  and the societal implications of generative AI technologies.   9. Omar Sanseviero BlogThe Omar Sanseviero Blog is an excellent resource for staying updated with the latest developments in large language models (LLMs) and machine learning (ML).   As a prominent machine learning engineer at Hugging Face  Omar brings a wealth of experience from his previous work at Google and his contributions to open-source projects. His blog covers a range of topics  including the latest releases and advancements in transformer models  multimodal models  and the integration of ML in various domains such as audio and computer vision.   Omars role at Hugging Face involves leading teams and initiatives that bridge open-source projects with cutting-edge research  making his insights particularly valuable for those interested in the practical applications and future directions of ML technology.   He also shares updates on collaborative projects and tools developed by the Hugging Face community  such as Hugging Face Spaces  which fosters community-driven ML demos and applications.   His blog is not only informative but also reflects his commitment to democratizing access to advanced ML tools and resources  making it a must-read for anyone keen on staying informed about the latest in LLM and ML research.   10. Lilian Weng BlogLilian Wengs blog  LilLog  is a great resource for keeping up to date with LLM research and news. OpenAI employee Lilian Weng documents her learning notes on her blog  which focuses on practical AI safety and alignment.   Her posts cover a wide range of topics related to AI and machine learning  including contrastive representation learning  diffusion models  neural architecture search  and reducing toxicity in language models.   Her writings have been praised by readers on LinkedIn  who have described her articles as insightful  systematic  and the most insightful  clear  and systematic they have ever seen. Weng also shares her blog posts on GitHub.   If you like the article and would like to support me  make sure to:U+1F44F Clap for the story (50 claps) to help this article be featuredSubscribe to To Data & Beyond NewsletterFollow me on MediumU+1F4F0 View more content on my medium profileU+1F514 Follow Me: LinkedIn U+007CYoutube U+007C GitHub U+007C TwitterSubscribe to my newsletter To Data & Beyond to get full and early access to my articles:To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   Are you looking to start a career in data science and AI and do not know how? I offer data science mentoring sessions and long-term career mentoring:Mentoring sessions: https://lnkd.in/dXeg3KPWLong-term mentoring: https://lnkd.in/dtdUYBrM


Reinforcement Learning: Introducing Deep Q* Networks  Part 6

You may have heard of Project Q*  a leaked idea from OpenAI in the year 2023 that is rumoured to represent a major breakthrough in the research for Artificial General Intelligence (AGI). While nobody knows what the project entails  I stumbled across an idea that is inspired by the name Q-star  by combining my previous knowledge in Q-Learning and my current foray into search algorithms  in particular the A* Search algorithm.   While I do not claim to have understood the meaning behind Project Q* (in fact  far from it)  this article reports a new model  which I will henceforth call the Deep Q* Networks  that has demonstrated a significant upgrade in efficiency to the vanilla Deep Q-Networks that is widely used in the field of Reinforcement Learning. This article represents a continuation (Part 6) of the series of explorations in Reinforcement Learning from scratch  and one can find the introductions of Q-Learning and Deep Q-Networks in the previous articles in the series here:   Reinforcement Learning: SARSA and Q-Learning  Part 3Introducing the Temporal Difference family of iterative techniques to solve the Markov Decision Processpub.towardsai.net   Reinforcement Learning: Function Approximation and Deep Q-Networks  Part 4Reinforcement Learning with continuous state spaces and gradient descent techniquespub.towardsai.net   1. The Analogy from A* Search AlgorithmOf note  the Deep Q-Networks applies the epsilon-greedy approach in training  which specifies a certain probability where the actions are executed completely at random so that the agent can explore the action-state space sufficiently.   Comparing this approach with the Search literature  this uniform randomness approach may be analogous to the Dijkstras or Breadth-First Search algorithm  where we trace a path from the starting point radially  in perhaps random directions  until the destination point is reached.   An upgrade to Dijkstras algorithm is the A* Search algorithm  which adds a heuristic map  which acts as a cost gradient  to guide the expansion of nodes most efficiently towards the goal. We can use a simple grid below as an illustration of the differences between the A* Search and Dijkstras algorithm. Note that in the simple grid  the 1 represents walls  while 0 represents possible paths.   # Dijkstra's Algorithm grid = [[S  1  0  0  0  0]  [0  1  0  0  0  0]  [0  1  0  0  0  0]  [0  1  0  0  0  0]  [0  0  0  0  0  G]] best_policy = [['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['>'  '>'  '>'  '>'  '>'  '*']] # -1 below represents the nodes that were not explored expansion_steps = [[0  -1  -1  -1  -1  -1]  [1  -1  12  -1  -1  -1]  [2  -1  9  13  -1  -1]  [3  -1  7  10  14  -1]  [4  5  6  8  11  15]]] # A* Algorithm heuristic_map = [[9  8  7  6  5  4]  [8  7  6  5  4  3]  [7  6  5  4  3  2]  [6  5  4  3  2  1]  [5  4  3  2  1  0]] best_policy = [['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['>'  '>'  '>'  '>'  '>'  '*']] expansion_steps = [[0  -1  -1  -1  -1  -1]  [1  -1  -1  -1  -1  -1]  [2  -1  -1  -1  -1  -1]  [3  -1  -1  -1  -1  -1]  [4  5  6  7  8  9]]From the short illustration above  we see that the A* algorithm took the most direct and efficient path to find the goal  while Dijkstras algorithm blindly expanded into the open space. This is because at each expansion step  the A* is guided by the value of the heuristic map  and the expansion always prioritizes expanding into the cell with the lowest heuristic value.   In the A* algorithm  the search becomes much faster but this also depends on the heuristic map that we craft  which must depend on our knowledge of the possible directions of the destination point relative to the start node.   2. Moving from A* Search to Q* LearningIn Deep Reinforcement Learning or Machine Learning in general  there is also a problem with the generalization of learning  which perhaps hinders the path towards Artificial General Intelligence (AGI). For instance  while a human being can logically differentiate amongst objects with perhaps few instructions  a supervised learning model may require thousands of training examples to be accurate enough in the differentiation.   In the context of Deep Reinforcement Learning  hundreds and thousands of episodes of training may need to be expended before the agent arrives at a good set of policy solutions. This poses another issue when we need to deploy an agent  perhaps a robot  which learns on the fly (after pretraining during simulation) in the real world  where the agent may break things if it erroneously executes a disastrous action.   To address this issue  we can port over the idea of a heuristic from A* Search to Q-Learning and Deep Q-Networks. This means that instead of relying on a blindly random exploration paradigm  we can alter our algorithm such that the exploration step is guided intelligently in the right direction  which is also in line with how humans naturally learn.   For instance  if a baby human knows that if he walks too fast  he may fall. If his goal is to walk steadily  naturally  he would not be suddenly jumping forward or attempting to run as his next experimentation. This is the logic behind an exploration paradigm guided by a reasonable heuristic.   To implement the Deep Q* Networks  I propose 3 main modifications to the DQN algorithm:   Normalize the Q-values from the Policy Network  which will then represent the probabilities of the respective actions by which the agent will act  based on the epsilon-greedy framework. This means that instead of taking complete random actions  the probability of the actions taken will be informed by the trained Q-values. This probability paradigm forms the heuristic that the agent will use to explore the action-state space.Allow human supervision in the early episodes will allow the good state-action pairs to be appended into the Replay Buffer. After the supervision ends  the independent agent will navigate the environment by itself and it will immediately experience failures because of the initial random weights. However  because of the good mix of state-action pairs in the Replay Buffer  the exploration heuristic is immediately enhanced  and the agent learns quickly.Using an Autoencoder architecture in both the Policy Network and Target Network has been observed to reduce overfitting and stabilize the training process. The Autoencoder applies an Unsupervised Learning approach to Supervised Learning allowing the network to better detect patterns and capture the overall trend. In a sense  this also mirrors how humans effectively learn  not only by memorizing knowledge through brute force (direct supervised learning) but also by self-organizing patterns in knowledge while they learn to understand and capture a bigger picture.With these above ideas in mind  let us now move on to transform the Deep Q-Networks into the Deep Q* Networks  and compare their distinct performances.   3. Deep Q* Networks  Modifications and ResultsSimilar to Part 4 of our Reinforcement Learning series  we will be using the Gymnasiums Lunar Lander environment  and I will encourage you to check it out (the link attached earlier) to better understand the requirements of the environment.   In short  the goal is to safely land the Lunar Lander on the Moons surface  as quickly as possible  without crashing. The environment is considered solved when the agent accumulates rewards of above 200 on average over 100 past episodes during training. After the training  the trained agent should also be evaluated over 100 episodes to validate its true performance.   A completely untrained Lunar Lander taking random actions probably crashes on every episode and looks like this below:   While a rigorously trained agent probably lands the Lunar Lander quite successfully and looks something like this:   Moving on we will now look at the modifications that I make. Instead of the TensorFlow Keras framework that I used in Part 4  in this article  we will explore the PyTorch implementation. In accordance with the 3 main modifications:   Normalize the Q-values from the Policy Networkclass DQNAgent: def __init__(self  input_dim  output_dim  gamma=0.99  lr=1e-3  tau=0.005): self.policy_network = DQN(input_dim  output_dim).float() self.target_network = DQN(input_dim  output_dim).float() self.target_network.load_state_dict(self.policy_network.state_dict()) self.lr = lr self.optimizer = optim.AdamW(self.policy_network.parameters()  lr=self.lr) self.gamma = gamma self.tau = tau def act(self  state  epsilon): state = torch.FloatTensor(state).unsqueeze(0) q_values = self.policy_network(state).detach() if np.random.rand() < epsilon: if np.random.rand() > epsilon: # Normalize Q-values to use as probabilities for exploration q_values -= q_values.min() # Shift Q-values to make them all positive q_values += 0.05 # Set a base probability value probs = q_values / q_values.sum() action = torch.multinomial(probs  1).item() else: action = np.random.randint(env.action_space.n) else: action = q_values.argmax().item() # Choose the best action based on Q-values return action # other class methods below Note that in the above implementation  we set a small Tau value of 0.005. This is critical to stabilize the steep learning curve that the Deep Q* Network will experience and the rewards will climb very fast. We also set a base Q-value of 0.05 such that the improbable actions would not get pushed to almost zero probability.   2. Allow human supervision in the early episodes   human = DQNAgent(env.observation_space.shape[0]  env.action_space.n  gamma=gamma  lr=lr  tau=tau) human.policy_network.load_state_dict(torch.load('dqn_agent_weights.pth')) for episode in range(episodes): state = env.reset() episode_reward = 0 done = False timestep = 0 while not done: # Use current epsilon in the act() method timestep += 1 if len(replay_buffer.buffer) < 10000: action = human.act(state  epsilon=0.5) else: action = agent.act(state  epsilon=epsilon) next_state  reward  done  _ = env.step(action) replay_buffer.add_to_buffer(state  action  reward  next_state  done) state = next_state episode_reward += reward if len(replay_buffer.buffer) > 10000: agent.train(replay_buffer  batch_size) # other codes belowWe used a perfectly trained agent in place of human supervision for our purpose. Of note  the loaded agent is trained with the (1) and (3) modifications  such that even when we set epsilon=0.5  it is taking probabilistic actions based on its trained Q-values. Hence  I observed that when epsilon  0.5  the agent still performs reasonably well. For the supervision process  I epsilon=0.5 to add uncertainty and variety to the agents actions  and it is observed to improve the performance.   3. Using an Autoencoder architecture   class DQN(nn.Module): def __init__(self  input_dim  output_dim): super(DQN  self).__init__() self.fc = nn.Sequential( nn.Linear(input_dim  64)  nn.ReLU()  nn.Linear(64  24)  nn.ReLU()  nn.Linear(24  64)  nn.Linear(64  output_dim) ) def forward(self  x): return self.fc(x)In the above simple Autoencoder architecture  notice a sharp bottleneck before the network propagates to the final outputs.   When the above modifications were applied  the Deep Q* Networks model is shown to converge much more quickly  stably and consistently compared with the vanilla DQN. In addition  the validation episodes from the Deep Q* Networks significantly outperform the vanilla DQN  with all episodes scoring above 200 rewards.   I also observe that modifications (1) and (2) contribute more critically to the efficiency gain  while modification (3) acts as a secondary advantage. Without either one of (1) and (2)  the speed of convergence quickly falls.   I illustrate the comparisons between the performances of Deep Q* Networks and the vanilla DQN below:   4. ConclusionWith the experimental results above  there is enough confidence to think that the Deep Q* Networks significantly outperform the vanilla Deep Q-Networks and that the idea of the trained exploration heuristic holds effective promise in improving training efficiency and outcomes.   In addition  the Deep Q* Network framework may allow better convergence of complex and tricky Reinforcement Learning tasks and environments  and this remains to be seen in future experimentations.   The problem of generalized learning and quick convergence in Deep Learning is an important field of research and may hold the key to Artificial General Intelligence (AGI). When we progress further in this field  hopefully  one day we may more confidently allow online learning for real-time robot agents  which would be much less likely to commit critical errors that endanger their environment and humans.   Finally  if you are interested in Deep Q-Networks extended to multiple agents  remember to check out the previous article in the series on Multi-Agent cooperation with DQN  which represents another fascinating field in Reinforcement Learning:   Reinforcement Learning: Multi-Agent Cooperation with MADQN Part 5Multi-agent reinforcement learning with 3 MADQN frameworks on the ma-gyms Switch4 environmentpub.towardsai.net   Congratulations on reaching the end of this research article! In Part 7 of this Reinforcement Learning series  we will be introducing the Policy Gradient methods  so stay tuned!   Thanks for reading! If you have enjoyed the content  pop by my other articles on Medium and follow me on LinkedIn.   Support me!  If you are not subscribed to Medium  and like my content  do consider supporting me by joining Medium via my referral link.   Join Medium with my referral link  Tan Pengshi AlvinAs a Medium member  a portion of your membership fee goes to writers you read  and you get full access to every storytanpengshi.medium.com


Fine-Tuning and Evaluating Large Language Models: Key Benchmarks and Metrics

In generative AI  we must first define the problem statement. Then  select a model accordingly. We must then select the model that best fits the specific task at hand. For example  we can use the FLAN-T5 model to summarize dialogues. We can also choose any other model. We then proceed with one  two  and more shots to see how it performs. If it does not produce the desired results  we may need to fine-tune the model. Then  well look at the evaluation part. In this post  we will go into greater detail about fine-tuning and evaluating the model.   In-context learning (where you give one  two  or more shots) has limitations for certain cases and does not work well for smaller models. In-context learning is a process in which you try zero shots  one shots  or multiple shots and provide examples to LLM in prompts so that the model can generate for an unknown prompt.   Fine-tuning is a supervised learning process that uses a dataset of labeled examples to update the LLMs weights. The labeled examples are prompt completion pairs  as illustrated in the diagram above. The fine-tuning process extends the models training to improve its ability to generate high-quality completions for a specific task.   For example  If I want to finetune the model to improve sentiment analysis capability  we would build up a dataset of examples that begin with the instruction Classify. We will build a dataset with many such example prompts as mentioned above.   Classify the following sentence into positive or negative: Text: {input_text} Summary: {expected sentiment}We can use many example prompts as our training dataset. This includes instruction to classify the text along with the associated labels.   For translation:   Translate this sentence to Spanish: English: {input_sentence} Spanish: {expected_translation}To summarize what we have said:   Use Pretrained Model: A model already trained on a large  general dataset.Task-Specific Examples: Prompt completion pairs specific to the desired task.Prepared Instruction Dataset Split: We divide the dataset into training  validation and test set.Finetuning Process: We fine-tune the model using training and validation datasets and then evaluate the performance on testset using cross-entropy loss.Surprisingly  good results can be obtained with relatively few examples. In comparison to the billions of pieces of text that the model saw during pre-training  only 5001 000 examples can consistently produce good results.   Drawbacks of finetuning on a single task:Catastrophic forgetting happens because the full fine-tuning process modifies the weights of the original LLM. While this leads to great performance on a single fine-tuning task  it can degrade performance on other tasks.How to avoid catastrophic Forgetting?Multi Task FinetuningCatastrophic Forgetting can be avoided by providing a variety of examples to the model. For example  we can provide examples of summarization prompts  translation prompts  and rating prompts. This requires numerous examples of each instruction when completed. The instruct version of the model is fine-tuned so that it can follow prompted instructions.   One example is the FLAN family of models. FLAN (fine-tuned language net) refers to a specific set of instructions used to fine-tune various models. Many models are based on FLAN models. For example  the FLAN T5 model is based on the FLAN model. SAMSUM is one of the datasets that FLAN T5 uses. There are several pre-trained FLAN T5 models that have been fine-tuned on SAMSUM  including Phil Schmid/flan-t5-base-samsum and jasonmcaffee/flan-t5-large-samsum on Hugging Face. If we want to fine-tune the FLAN T5 model specifically for formal dialogue conversations  we can do so using the DIALOGUESUM dataset.   Models fine-tuned on DialogSum can be applied to areas like customer support  meeting minutes generation  chatbot summarization  and more.   2. PEFT (Parameter efficient fine tuning)Training LLMs is computationally intensive. Full finetuning is computationally expensive as it might change each weight in the model. First  we start with a pretrained LLM like GPT-3. This model already has a vast amount of knowledge and understanding of language. Then we provide task-specific datasets  which could be data for question answering or sentiment analysis or any other customer dataset. During training  full finetuning process makes slight adjustments to every weight in the pretrained model. While the model weights are substantial  we have other important aspects during training like Optimizer  which adds up to the cost. For example  Optimizer States  gradients  forward activation  and temporary memory. These additional components add up to the training cost.   Three main approaches are used in PEFT: Selective / reparameterization/additive.   1. SelectiveHere  we select a subset of initial LLM parameters to fine-tune.   2. ReparameterizationWe reparameterize model weights using a low-rank representation. We will discuss LoRA in detail below.   LORA: Low Rank Representation:   Each layer in a transformer architecture has multiple weight matrices for different operations  like self-attention or feed-forward networks. These matrices can have different sizes depending on the specific layer and configuration. Let us take an example by picking a matrix of size 512 x 64 = 32 768 parameters. Let us now see LoRA with rank = 8.   Original Weight Matrix: Dimensions: 512 x 64  Parameters: 32 768 (512 x 64)Matrix A (Rank Decomposition): Dimensions: 8 x 64 (rank x original dimension)  Parameters: 512 (8 x 64)Matrix B (Rank Decomposition): Dimensions: 8 x 512 (rank x original dimension)  Parameters: 4 096 (8 x 512)Total LORA Parameters: 512 (A) + 4 096 (B) = 4 608Approximation:   The original weight matrix (W) is approximated by the product of A and B:   Z  W  A * B   Reasoning Behind the Dimensions:   The dimensions of A and B are chosen to capture the essence of the original weight matrix (W) with fewer parameters.The rank (here  8) controls the trade-off between efficiency and accuracy. A lower rank leads to fewer parameters but might result in a slightly less accurate approximation.We can also create task-specific decomposition matrices.In the example we discussed  LORA achieves a reduction of approximately 86% in the number of trainable parameters needed for fine-tuning. Heres the summary.   Original Weight Matrix: 32 768 parameters (512 x 64)Total LORA Parameters: 4 608 parameters (512 + 4 096)3. AdditiveWe add trainable layers or parameters to the model in the form of adapter modules.   The two main additive approaches are:   Adapter Modules: These are small  trainable neural network modules strategically inserted into specific layers of the pre-trained LLM. They help the LLM learn task-specific information without drastically changing its underlying knowledge.Prompt Tuning: This approach doesnt involve adding any new modules to the model itself. Instead  it focuses on crafting specific prompts (essentially instructions or questions) that guide the pre-trained LLM toward the desired task.All these approaches are similar to transfer learning  but they are more efficient in that they only fine-tune a subset of parameters rather than fine-tuning the complete layer. Even adapter modules are lightweight.   PEFT is particularly beneficial when dealing with large LLMs that have billions or even trillions of parameters  as fine-tuning all of them can be computationally expensive and resource-intensive.   PEFT is less prone to the catastrophic forgetting problems of full fine-tuning. Full fine-tuning results in a new version of the model for every task you train on.   Metrics to assess the performanceIn the language model  evaluation is more challenging since the output is non deterministic.   Let us explore some of the metrics that we can use to evaluate.   ROUGE-1: (Recall-Oriented Understudy for Gisting Evaluation)ROUGE-1 is recall oriented metric  which means it prioritizes identifying how many of the important words from the reference summaries are included in the generated summary. ROUGE 1 focuses on individual words(unigrams). Similarly  ROUGE-2 focuses on bigrams and so goes on.   Let us take an example of ROUGE-1:   Lets walk through an example step-by-step:   Reference Text: Mike really loves drinking tea.Generated Text: Mike adores sipping tea.Step 1: Identify Unigrams   Reference Text Unigrams: {Mike  really  loves  drinking  tea}Generated Text Unigrams: {Mike  adores  sipping  tea}Step 2: Count Overlapping Unigrams   Overlapping Unigrams: {Mike  tea}Number of Overlapping Unigrams: 2ROUGE-1 Recall   ROUGE-1 Precision   ROUGE-1 F1 Score   ROUGE-L:ROUGE-L is a metric used to evaluate the quality of text by measuring the longest common subsequence (LCS) between a generated text and a reference text. The LCS takes into account order of words making it more sensitive to the overall structure of the text compared to simple n gram overlap.   Lets walk through an example step-by-step:   Reference Text: It is cold outside (We can see two subsequence It is in italics and cold outside in bold.)Generated Text: It is very cold outside (We can see two subsequence It is in italic and cold outside in bold.)ROUGE-L Recall = LCS(Gen  Ref) / unigrams in reference = 2/4 = 0.5`   ROUGE-L Precision = 2 / 5 = 0.4   ROUGE-L F1 = 2 . (0.2/0.9) = 0.44   ROUGE ClipingROUGE sometimes give misleading results. Let us explore this:   Example 1: Repetitive Generated Text   Reference (human): The sun is shining brightly.Generated output: shining shining shining shiningWithout clipping:   Unigram Matches: shining (matches four times)ROUGE-1 Precision: 4/4 = 1.0This perfect score is misleading because the generated text is repetitive and lacks meaningful content.   With clipping:   Clipped Unigram Matches: shining (matches only once  as in the reference)Modified Precision: 1/4Clipping provides a more accurate reflection of the generated texts quality.   Example 2: Reordered Generated Text   Reference (human): The sun is shining brightly.Generated output: brightly the sun is shiningWith clipping:   Clipped Unigram Matches: The  sun  is  shining  brightly (matches exactly as in the reference)Modified Precision: 5/5=1Despite the different word order  clipping correctly identifies that the generated text includes all relevant unigrams in the correct frequency  giving it a perfect score. This could also be misleading.   To sumup  ROUGE clipping improves evaluation accuracy by limiting unigram matches to the count present in the reference text  preventing artificially inflated scores from repetitive words  and correctly handling word order variations.   BLEUBLEU primarily focuses on n-gram precision  which means it counts how often sequences of words (n-grams) in the machine translation match those in the reference translations. It considers 1-grams (single words)  2-grams (phrases)  etc. You can also refer to it as average precision across range of n-gram sizes.   Other Metrics and BenchmarksThere are other important metrics also used for evaluation  which are listed below in table:   With regards to HELM  One important feature of HELM is that it assesses on metrics beyond basic accuracy measures  like precision of the F1 score. The benchmark also includes metrics for fairness  bias  and toxicity  which are becoming increasingly important to assess as LLMs become more capable of human-like language generation  and in turn  of exhibiting potentially harmful behavior. HELM is a living benchmark that aims to continuously evolve with the addition of new scenarios  metrics  and models.   ConclusionIn this post  we saw an important aspect of fine-tuning a large language model. We started by discussing zero shot  one shot  two shot  more shot to see if the model works by generating the correct output. If it does not  we need to finetune the model. We can finetune the model by picking a relevant model based on our task requirement. Then  we finetune the model by giving it more examples along with labels. We also saw how finetuning the model can lead to catastrophic forgetting and the way to avoid it is to finetune on multiple tasks so that the model generalizes well. In addition  we can also use Parameter-efficient fine-tuning  where we discussed 3 techniques to avoid computational problems as well. Techniques like LoRA is very beneficial. We then moved towards evaluating the model where we studied some important metrics like ROUGE  BLEU  and other benchmarks available.   References[1] https://cobusgreyling.medium.com/catastrophic-forgetting-in-llms-bf345760e6e2   [2] https://arxiv.org/html/2401.05605v1   [3] https://www.linkedin.com/pulse/catastrophic-forgetting-side-effect-fine-tuning-large-karan-sehgal-jjkqe/   [4] https://medium.com/@sthanikamsanthosh1994/understanding-bleu-and-rouge-score-for-nlp-evaluation-1ab334ecadcb   [4] https://www.deeplearning.ai/courses/generative-ai-with-llms/