GraphRAG + GPT-4o-Mini is the RAG Heaven

Disclaimer: This implementation of GraphRAG is inspired by the paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization by Darren Edge et. al. The code is not entirely similar to the papers codebase  though the prompts for certain tasks are taken from the papers codebase.   This is the second blog in a multi-part blog series series about GraphRAG. In this blog series  our goal is to achieve the following    Understand the fundamentals of GraphRAGThe need for GraphRAG: GraphRAG vs. Semantic/Keyword-based RAGImplement GraphRAG components from scratch in PythonApply GraphRAG for Content-Based Movie Recommendation: GraphRAG4ReccomendationUse GPT-4o-Mini for creating the graph and providing recommendationsWe will achieve the following output by the end of this multi-part blog series.   The following is the GitHub repository for the GraphRAG4Rec codebase.   A naive implementation of GraphRAG for Movie Recommendation on IMDB Top 1000 movies dataset. github.com   Other PartsPart 1: Introduction to GraphRAGPart 3: Extract  entities  relations  and claims to build the graph (coming soon)Part 4: Batch communities and prepare summarization reports (coming soon)Part 5: Query processing and recommendation generation via map-reduce prompting (coming soon)In this blog Well quickly understand the need for a graph-based retrieval augmented generation (GraphRAG) approach. Well compare this approach with a semantic or keyword-based RAG approach.   Understanding Semantic/Keyword-based RAGIn Semantic/Keyword-based RAG we combine the traditional information retrieval strategies with language generation to produce more accurate and contextually relevant responses.   Components of Semantic/Keyword-based RAGThe following are the components of a semantic/keyword-based RAG.   A Document Corpus is a collection of texts or documents that serve as the knowledge base.The embedding model converts text into vector representations that capture semantic meaning.A vector database that stores and indexes the embedded representation of documents.The retriever finds relevant documents based on the query.A (Large) Language Model to generate responses based on the retrieved information and the query.The following flow represents the traditional RAG (semantic/keyword-based) process.   Now  we wont go too deep into the details of chunking strategies or retrieval strategies like query decomposition  re-ranking  etc. These things do help augment the quality of the final output.   We now understand the fundamentals of GraphRAG and traditional RAG (semantic/keyword-based) along with the components of the respective approaches. Now its time to compare these approaches with an example.   Well use the same movie scenario and hypothetically compare both approaches.   ComparisonWell compare the approaches on the following points.   Knowledge representationRetrieval mechanismContext understandingScalabilityQuery interpretationInformation synthesis1. Knowledge representationIn GraphRAG  we represent movies  actors  directors  and themes as interconnected entities. For example  The Matrix is connected to the sci-fi and Action genres  The Wachowskis as directors  and Keanu Reeves as an actor.In traditional RAG  we might store The Matrix as a text chunk: The Matrix is a 1999 science fiction action film directed by the Wachowskis  starring Keanu Reeves.The advantage of GraphRAG is it can easily answer questions like What other sci-fi movies star actors from The Matrix? by traversing the graph.   Traditional RAG would struggle with this without specific text mentioning such connections. To make such traditional RAG work with such queries we might need to implement some kind of query decomposition and dependency planning.   2. Retrieval mechanisms   In GraphRAG  if we have a query like sci-fi dystopian movies  the retrieval can start from communities similar to or with the Sci-Fi node  and traverse to a more local community with a node like dystopian and end up returning the movie  The Matrix.While in traditional RAG  if no chunks mention sci-fi or dystopian along with the movie The Matrix or some other movie then the output might be very generic i.e. related to the keyword sci-fi or might have some movie whose theme is dystopian (mentioned in the chunk) but is not a sci-fi.Thus  GraphRAG can find relevant content even if query terms dont exactly match the stored text.   3. Context understanding   GraphRAG can understand that Inception and The Matrix are related because they share the sci-fi genre and mind-bending concepts theme  even if thats not explicitly mentioned in any text chunk.Traditional RAG might not be able to connect these two movies unless theres a specific text chunk comparing them.In this case  the relationship between the two movies  Inception and The Matrix  is implied via the genre and the theme that these movies share. And in the graph  there will be a connection between these two movies and they might even form a community. Thus  GraphRAGs implicit context understanding can help with more insightful recommendations.   4. Scalability   As our movie database grows  the hierarchical structure (C0  C1  C2) in GraphRAG allows for efficient navigation. We can quickly narrow it down from Movies to Sci-Fi & Action to Pure Sci-Fi Action. This also depends on how were designing our retriever  entity-based or via map-reduce over community reports.In the case of traditional RAG  we might struggle when answering broad queries as there can be a lot of unrelated but matching chunks similar to various parts of the query. We would then need to introduce re-ranking to filter suck chunks.Thus  GraphRAG can handle large  complex datasets more efficiently  especially for exploratory queries.   5. Query interpretation   For a query like movies like Inception but more action-focused  GraphRAG  via map-reduce over community reports  can understand that it needs to look for movies in the Sci-Fi Thriller category but closer to the Pure Sci-Fi Action category  potentially suggesting The Matrix.For the same query  the traditional RAG might struggle to capture the nuance of the query and might return movies mentioning both Inception and action.Thus  GraphRAG can handle more nuanced  context-dependent queries.   6. Information synthesis   For a query about the evolution of sci-fi movies from the 90s to 2010s  GraphRAG via map-reduce over community reports can collect information related to sci-fi movies and their release years. And then effectively use this information to answer such a broad question.Traditional RAG might get chunks similar to sci-fi  90s  or 2010s but struggle to thread the evolution narrative.With the ability to traverse over related entities  GraphRAG can provide more comprehensive  synthesized responses for complex queries.   No one size fits allWhile GraphRAG is a better approach for answering more nuanced  broad  and exploratory questions there are various use cases where traditional RAG is a better fit.   GraphRAG is very expensive  both in terms of the amount of tokens  embedding  and retrieval times.   If GraphRAG is being used with LLMs on the local system then the cost factor is a non-issue but still the indexing (extracting + embedding) time is quite high compared to computing the embeddings of the document chunks.   Traditional RAG is still a better choice for:   Simple  fact-based queries: For questions like What year was The Matrix released?  traditional RAG will be faster and more straightforward.Easier implementation: For smaller datasets or simpler use cases  traditional RAG is likely easier to set up and maintain.The reason for implementing GraphRAG for the content-based movie recommendation use case is simple and is already explained by the different query examples in the comparison above.   We want our RAG approach to answer highly broad (global)  nuanced (local)  and complex queries. Using a traditional RAG approach it's quite hard to consistently cater to such a broad range of queries.   ConclusionWhile GraphRAG offers significant advantages in understanding the context  relationships  and complex queries in our movie domain  traditional RAG still has its place  especially for simpler  more straightforward use cases.   From the next blog onwards well start with the implementation of the key components of GraphRAG in Python. Later  combine all of the components to recommend movies based on a user query.


Disclaimer: This implementation of GraphRAG is inspired by the paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization by Darren Edge et. al. The code is not entirely similar to the papers codebase  though the prompts for certain tasks are taken from the papers codebase.   This is the first blog in a multi-part blog series series about GraphRAG. In this blog series  our goal is to achieve the following    Understand the fundamentals of GraphRAGThe need for GraphRAG: GraphRAG vs. Semantic/Keyword-based RAGImplement GraphRAG components from scratch in PythonApply GraphRAG for Content-Based Movie Recommendation: GraphRAG4ReccomendationUse GPT-4o-Mini for creating the graph and providing recommendationsWe will achieve the following output by the end of this multi-part blog series.   The following is the GitHub repository for the GraphRAG4Rec codebase.   GitHub - vatsalsaglani/GraphRAG4Rec: A naive implementation of GraphRAG for Movie Recommendation onA naive implementation of GraphRAG for Movie Recommendation on IMDB Top 1000 movies dataset. github.com   Other PartsPart 2: GraphRAG vs Semantic/keyword-based RAGPart 3: Extract  entities  relations  and claims to build the graph (coming soon)Part 4: Batch communities and prepare summarization reports (coming soon)Part 5: Query processing and recommendation generation via map-reduce prompting (coming soon)In this blog Well understand the fundamentals of GraphRAG with an example.   What is GraphRAG?GraphRAG is an advanced Graph-based Retrieval Augmented Generation (GraphRAG) approach introduced in the paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization by Darren Edge et. al. This approach combines graph theory  information retrieval  and LLMs.   The core concept is that the entities in our text are represented as nodes in the graphs and the relations between these entities represent the edges between the nodes. The graph is then hierarchically divided into communities and summarized into community reports.   At query time  weve to decide how deep should we explore the communities to find relevant communities. The more the depth the more the computations/LLM calls.   Once the relevant communities are retrieved we can answer the user query using the summary reports of those communities.   The following diagram depicts the entire process.   Key components of GraphRAGAs shown in the above image  weve divided the GraphRAG process into the following three key components.   ExtractEmbedQueryLets understand these components individually.   ExtractIn the extract component  we do the following things    Extract EntitiesExtract Relations among EntitiesExtract Claims on EntitiesExtract Claims on RelationsWell understand this with an example.   Suppose we have the following text.   Movie: The Shawshank Redemption Two imprisoned men bond over a number of years  finding solace and eventual redemption through acts of common decency.\nYear: 1994\nDirector: Frank Darabont\nCast: [Tim Robbins  Morgan Freeman  Bob Gunton  William Sadler]\nCertificate: A   Step 1: Extract Entities   The Shawshank Redemption (Movie)Frank Darabont (Person  Director)Tim Robbins (Person  Actor)Morgan Freeman (Person  Actor)Bob Gunton (Person  Actor)William Sadler (Person  Actor)1994 (Year)A (Certificate)Step 2: Extract Relations   The Shawshank Redemption  directed by  Frank DarabontThe Shawshank Redemption  stars  Tim RobbinsThe Shawshank Redemption  stars  Morgan FreemanThe Shawshank Redemption  stars  Bob GuntonThe Shawshank Redemption  stars  William SadlerThe Shawshank Redemption  released in  1994The Shawshank Redemption  has certificate  ASteps 34: Extract Claims for Entities and Relations   The Shawshank Redemption: Two imprisoned men bond over a number of years  finding solace and eventual redemption through acts of common decency.The Shawshank Redemption: Released in 1994The Shawshank Redemption: Has certificate AThe central node will be the name of the movie  The Shawshank Redemption.   If we were to plot the entities  relations  and claims it would look something like the image below.   EmbedAfter processing the required steps for the first component  Extract  for all the documents  the extracted information will be embedded into a graph.   As we need to embed movies into a graph lets take two more movie texts.   Movie: Inception\nGenre: [Sci-Fi  Action  Thriller]\nYear: 2010\nDirector: Christopher Nolan\nCast: [Leonardo DiCaprio  Joseph Gordon-Levitt]\nCertificate: PG-13   Movie: The Matrix\nGenre: [Sci-Fi  Action]\nYear: 1999\nDirector: The Wachowskis\nCast: [Keanu Reeves  Laurence Fishburne]\nCertificate: R   Extract output for Inception   Step 1: Extract Entities   Inception (Movie)Christopher Nolan (Person  Director)Leonardo DiCaprio (Person  Actor)Joseph Gordon-Levitt (Person  Actor)2010 (Year)PG-13 (Certificate)Sci-Fi (Genre)Action (Genre)Thriller (Genre)Step 2: Extract Relations   Inception  directed by  Christopher NolanInception  stars  Leonardo DiCaprioInception  stars  Joseph Gordon-LevittInception  released in  2010Inception  has certificate  PG-13Inception  has genre  Sci-FiInception  has genre  ActionInception  has genre  ThrillerSteps 34: Extract claims on Entities and Relations   Inception: A skilled thief with the rare ability to extract information from peoples minds is offered a chance to regain his old life as payment for a task considered to be impossible: inception  the implantation of another persons idea into a targets subconscious.Inception: Released in 2010Inception: Has certificate PG-13Inception: Is a Sci-Fi filmInception: Is an Action filmInception: Is a Thriller filmExtract output for The Matrix   Step 1: Extract Entities   The Matrix (Movie)The Wachowskis (Person  Directors)Keanu Reeves (Person  Actor)Laurence Fishburne (Person  Actor)1999 (Year)R (Certificate)Sci-Fi (Genre)Action (Genre)Step 2: Extract Relations   The Matrix  directed by  The WachowskisThe Matrix  stars  Keanu ReevesThe Matrix  stars  Laurence FishburneThe Matrix  released in  1999The Matrix  has certificate  RThe Matrix  has genre  Sci-FiThe Matrix  has genre  ActionSteps 34: Extract claims on Entities and Relations   The Matrix: A computer programmer discovers that reality as he knows it is a simulation created by machines to subjugate humanity  and joins a rebellion to overthrow them.The Matrix: Released in 1999The Matrix: Has certificate RThe Matrix: Is a Sci-Fi filmThe Matrix: Is an Action filmEmbed Step 1: Create a Graph   Now that we have the entities  relations  and claims from all three movies we can embed these into a graph like the following.   Embed Step 23: Detect Communities and Establish Hierarchy   We can divide the graph into the following two communities based on the genres.   Drama and Crime CommunitySci-Fi and Action CommunityWe can use a hierarchical community detection algorithm  the Leiden Algorithm  to cluster the nodes into separate communities.   First  lets look at how the hierarchical communities will come out.   We have the following hierarchies.   C0  Movies: This community contains all the movies in our dataset. It represents a diverse range of movies spanning across different genres  periods  and themes. The movies share common elements such as directors  actors  genres  and release year  but differ in their specific content and style.C1  Drama and Crime: This community focuses on dramatic storytelling with elements of crime.C1  Sci-Fi and Action: This community combines elements of science fiction and action.C2  Pure Sci-Fiction: This sub-community  represented by The Matrix  focuses on science fiction concepts with a heavy emphasis on action.C2  Sci-Fi Thriller: This sub-community  represented by Inception  combines science fiction elements with psychological thriller aspects.With this hierarchy  we have both global and local categorization. The C0 and C1 clusters/groups/communities are very broad  global  and the C2 cluster/group/community are very specific  local.   Embed Step 4: Summarize Communities   C1  Drama and CrimeIntense character-driven narrativesExploration of human relationships and emotionsThemes of justice  redemption  and perseveranceRealistic portrayals of criminal justice systems2. C1  Sci-Fi and Action   Futuristic or alternative reality settingsMind-bending concepts and technologiesBlend of intellectual stimulation and visual spectacleExploration of the nature of reality and consciousness3. C2  Pure Sci-Fi Action   Dystopian future settingsAdvanced technology as a central plot elementHigh-octane action sequencesThemes of human vs. machine4. C2  Sci-Fi Thriller   Complex  layered narrativesPsychological manipulation and explorationBlurring lines between reality and imaginationIntellectual puzzles and mind-bending conceptsSummary reports can also contain awards  performance by actor directory  box office results  etc. as well.   Well take a short detour and understand the Leiden algorithm with the movie example.   About Leiden AlgorithmThe Leiden algorithm is an improved version of the Louvain method for community detection. It works by optimizing modularity  a measure of the density of links inside communities compared to links between communities.   First  lets understand modularity.   Modularity is a measure of how well a network is divided into communities. We can think of it as    High modularity means there are many connections within communities and few connections between different communities.Low modularity means connections are more evenly spread  with no clear community structure.For our movie example  high modularity would mean movies within a community have many shared characteristics like the Sci-Fi and Action community. Low modularity means fewer characteristics in common like the Drama and Crime community.   Hierarchical Community Detection StepsLets look at the community detection steps for the movie example.   Step 1: Start with individual nodes   Begin with each movie as its own community.   Community 1: The Shawshank RedemptionCommunity 2: InceptionCommunity 3: The MatrixStep 2: Merge nodes into communities   Look at the connections between movies like shared genres or themes and merge them if it improves modularity.   Merge Inception and The Matrix into a Sci-Fi and Action community.The Shawshank Redemption remains in its own Drama and Crime community.Step 3: Create the first level of hierarchy (C1):   C1 Community 1: Drama & Crime (The Shawshank Redemption)C1 Community 2: Sci-Fi & Action (Inception  The Matrix)Step 4: Communities as Nodes   Now consider communities as nodes.   Step 5: Repeat Steps 1  2  and 3 at a higher level   Look for connections between these community nodes. In this case  there arent enough communities to merge further  so we shop here for the C0 level.   Step 6: Refine lower levels   Go back to the Sci-Fi and Action community and look for subcommunities.   Split Inception and The Matrix based on their more specific characteristics.Step 7: Create the second level of hierarchy (C2)   C2 Community 1: Pure Sci-Fi Action (The Matrix)C2 Community 2: Sci-Fi Thriller (Inception)Finally  we have the following hierarchy.   QueryIn the query part  we use a map-reduce approach to find relevant communities using a map operation. The map outputs are then provided to the reduce (reducer) to answer the user query.   Lets look at the query process with an example query  I want to watch a crime drama.   The following is how the entire process will look like.   Map Phase   We first have the map phase. Here  every community report is passed to the mapper which will output how relevant the community is for the given query along with the movies.   The output of the map phase through every community will look like the following.   Drama and Crime C1:{ community: Drama & Crime C1  relevance_score: 95  movies: [The Shawshank Redemption]  reason: Directly matches the crime drama genre request }Sci-Fi and Action C1{ community: Sci-Fi & Action C1  relevance_score: 10  movies: [Inception  The Matrix]  reason: Does not match the crime drama genre request }Pure Sci-Fi Action C2{ community: Pure Sci-Fi Action C2  relevance_score: 5  movies: [The Matrix]  reason: Does not match the crime drama genre request }Sci-Fi Thriller C2{ community: Sci-Fi Thriller C2  relevance_score: 15  movies: [Inception]  reason: Has some thriller elements but not crime drama }Reduce Phase   The outputs from the map phase are passed to the reducer along with the user query to get a list of relevant suggestions along with other recommendations.   The following is how the output of the reduce phase will look like.   { relevant_communities: [ { community: Drama & Crime C1  relevance_score: 95  movies: [The Shawshank Redemption] } ]  other_suggestions: [ { community: Sci-Fi Thriller C2  relevance_score: 15  movies: [Inception] } ] }Moreover  we can communicate this output via an LLM by providing the user query and the relevant communities with movie details along with suggestions. We can prompt the LLM to personalize the output based on the user query and the relevant movies and extra suggestions.   ConclusionIn this blog  we got an introduction to GrapRAG and the key components  extract  embed  and query. Along with that we also learnt about hierarchical community detection using the Leiden algorithm. In the upcoming blogs  well build upon this knowledge to develop a GraphRAG module for a content-based movie recommendation system  GraphRAG4Recommendation.   See you in Part 2: GraphRAG vs semantic/keyword-based RAG.


6 Years of Studying ML in 16 Minutes

I have been studying machine learning for the past 6 years  in which I worked as an ML student researcher for over 2 years  and have even written my first 3 papers.   My journey started studying computer engineering  not knowing what ML was or even that it existed  to where I am now  soon joining my favorite AI startup as a research scientist!   In this blog post  I will be sharing my experience over these 6 years and explain what I did each year to get where I am now.   Things like what to expect for the first few years  what I did to get my first ML student roles  and most importantly  what you should be avoiding!   And trust me  there is a lot   Year 1Okay  before we get to years one and two  how did I get into tech?   Well  young Boris liked physics and math in high school and thought    Hmm  with physics you cant really make money  so I need to do engineering  i.e. applied physics.   Building a robot would be really cool!   But then I also need know how to program the robot to make it do the stuff I want it to do!   At that time  I didnt know AI and ML existed  but those were my thoughts.   This led me to study Computer Engineering at the TU Berlin!   The first two years were really tough  of course.   I had to take the standard courses like linear algebra 1  calculus 1 and 2  and a course on differential equations!   Luckily  I genuinely enjoyed learning math! But that doesnt mean it was easy for me.   In the beginning  it all doesnt make much sense and you dont know why you are learning all these mathematical formulas and abstract concepts.   But I promise you  at some point most of them will make sense and you will learn to appreciate and make use of them! Especially when learning ML!   The basics of these math skills will be the fundamentals you will later need for ML and give you an intuition for how to look at ML models in a mathematical sense.   But back then  again  I didnt even know AI excited!   I had a lot of electrical engineering and even some physics courses! Those were tougher!   But I also had my first CS courses and learned how to code in C!   Yes  in C  thats right! Remember  I was a computer engineering major  so my program was designed for low-level coding and electrical engineering.   However  I still had the standard course on data structure and algorithms in Java and also a course on theoretical CS.   All that is pretty much the standard things you learn when getting into a CS-related program.   Some CS theory and a lot of coding.   Besides normal college courses  I landed my first student researcher job at an optical physics lab about 6 months into my first year!   I wanted to somehow boost my resume  earn some money to survive college  and also just learn more stuff!   I then saw this listing at a research institute directly next to my uni and applied.   It was honestly quite surprising that they invited me to an interview because I literally had not much to offer except basic programming skills and a basic understanding of electrical engineering but I guess for the job I was supposed to do it was enough!   I was responsible for running a lot of experiments with optical fibers and doing measurements.   When starting a new job or project the learning curve will likely be very steep. Which is amazing! I learned a lot!   But  if you do the same measurements for over 1.5 years  the learning curve plateaus  and the job becomes boring.   In total  I stayed at this job for 3 years and this learning curve was completely flat after perhaps 89 months (if not less).   And this was a big mistake I really should have at least changed to a different team at this research institute after a year!   But I was quite exhausted  for these first 2 years  I slept 6 hours a night  didnt do much sports  and just worked a lot!   Which is normal  and I dont want to complain. I had a lot of fun! In fact  I am happy and proud I did all that!   But yeah  all of this happened in my first two years of uni!   Most importantly  I learned the basics of math and computer science and worked as a student researcher.   All of which helped me with studying ML  without even knowing ML existed!   Year 3I finally got into the third year  semesters 5 and 6  where I could choose some of my courses myself!   This fifth semester is where I saw that AI courses existed at my uni and is where I chose my very first AI course!   This is where the ML journey really started.   That said  this AI course was split into two parts  the first one was about old-school AI  NOT ML!   Yes  AI does not necessarily mean ML.   If you have an algorithm with a set of rules to make decisions  its effectively AI.   I learned about things like the STRIPS method. Looking back  its not that exciting honestly  but that is where I started and back then  I thought it was decently cool.   But the second half of this course was REALLY cool!   The second half was about reinforcement learning!   Which  in retrospect is a weird start into ML learning about RL before even knowing what a neural network was.   But maybe this is a good way to show you that it does not really matter how you start. If you keep going  you will learn all the fundamentals anyway. Just in a different order perhaps.   But I would still not recommend it  if you have the option to choose  but you get the point.   Anyway  I learned about things like bandit theory  MTCS  Markov Decision Processes  and finally RL algorithms such as Q-Learning!   So  in my fifth semester  2.5 years into college  there was still not that much ML  but these RL lectures really got me interested in ML  especially RL!   Thats why I wanted to do my bachelor's thesis in RL!   Which is what I did in my 6th semester! I worked on Deep RL for autonomous robotic navigation.   This was a complete cold start into DL! I didnt even know what a Neural Network was! I had to learn all of that on my own through YouTube videos and blog posts.   Even worse  in the beginning  I struggled a lot to even get the hardware set up!   And when I reached out to my supervisor for help  he said he thought I might not be ready for this thesis and I had 2 weeks to prove him otherwise.   And if I failed  he would have to drop the thesis with me   Which would have been so bad The semester had already started and I then would have to look for a completely new one.   But I pushed through and made it!   This thesis project was a loooooot of work!  A lot of engineering work and no real training itself  since the thesis was more on the deployment side of DRL agents than the training side.   Nevertheless  I still learned a lot of core coding skills  like debugging  and did get to learn PyTorch for the first time!   So  my final bachelor year was still a slow step into the world of ML  but a very firm one.   One that set the path to going all in on ML!   Which is why I then switched to pure a CS master!   Year 4So  my fourth year began and I went all in on ML! I selected only ML courses and projects!   But this of course came with a lot of challenges!   In my first graduate semester  I pretty much had one big course and one big project.   For the project  I continued to work on the same team for autonomous robotic navigation that I worked with during my bachelor thesis!   The project was still more of an engineering effort because we built a benchmarking suite for autonomous robots which again came with a lot of failing and debugging.   But this time I could focus a lot more on training our own agents using PyTorch and had to start reading papers to learn about things like PPO!   Of course  the beginning of reading papers is always a bit tough  because you have to get used to the lingo. But it felt so cool! I felt like a real scientist haha   The really cool thing was  that later that year  we actually published this work to one of the two best robotics conferences  IROS!!   That was so huge for me!   It was my first ML paper and it was even published at a top conference :)   Now  alongside this project  I had my first real ML course!   I learned all the basics of classical ML  e.g. what is supervised learning  unsupervised learning  what is the bias-variance tradeoff.   What are methods like Linear regression  Decision trees  Support vector machines  K-means  PCA  Boosting  and Ensemble learning?   And I learned about the basics of Neural Networks like what loss functions  gradient descent  backpropagation  and regularization are.   Alongside each lecture  there of course were practical homework assignments to implement the ideas we learned during the lecture. And those were again using PyTorch!   Now  besides uni  I still had this boring physics lab job. At this point  I was working there for 22.5 years already   But the cool thing was  the research institute I worked at also had an AI department!!!   So I wanted to internally switch teams!   I applied got an interview!  And was rejected   I mean  I get it. I was just starting my first real ML course and had no theoretical knowledge of any of the ML fundamentals.   So  I tried again *half a year later*  after completing the ML course and having gathered more basic PyTorch experience. And I then actually did get the job!   What an amazing start to my second semester  the second half of my fourth year!   I started my work as an applied scientist student researcher in the ML department!   I again had a steep learning curve and was so excited to get to work!   During these first 6 months  I started working on a lot of data engineering mainly using pandas  which I have never used before.   I learned a lot there!   And at uni  I also focused on purely practical learning!   I took two project courses.   I again continued to work on this robotics project. But  at this point  I felt a bit more of a fatigue working on the project.   It wasnt that exciting anymore  but it still a lot of work  and my learning curve plateaued. However  I continued to work on it because I hoped for another paper.   Nevertheless  I started to look at other cool ML domains and took another project course.   A project on a CV for medical image analysis!   This was my first CV project and I had to detect aneurysms in 3D images of the brain.   It was really cool! But I have never dealt with CV before and had never learned what a convolutional neural network was!   So the learning curve was again very steep!   I had to learn all of that knowledge myself by watching YT videos  and reading more papers!   In the end  the final project was not the worst  but also not the best either.   At least  looking back at it now.   And I think this is a good thing! If you are looking back at old projects and think they are bad  because you would have done things differently with your current knowledge  then you have gotten better!   So  yeah. This year was packed with all the ML I could fit in!   Most of it was actually working on ML projects and only taking one ML lecture! But a really important one.   So far  it was quite straightforward  but in the next year  I had to make some important decisions!   Year 5Now  uni continued as usual  but career-wise  I had to make those important decisions.   In my third graduate semester  I again took 1 lecture course and again 2 more projects!   I took my first actual Deep Learning course  which had a decent overlap with my first ML course.   I again learned about the same fundamentals of neural networks  but now also had lectures on CNNs  recurrent NNs  Autoencoders  and a bit of explainable AI.   So  nothing toooo crazy.   At this point  I am really into AI myself and I started watching paper review videos on YouTube and reading random papers on my own!   Perhaps because this course didnt have too much new stuff  and my job didnt teach much theoretical content  as well.   But  anyway  this habit of reading papers and learning stuff on my own are things I still do to this day and that I genuinely enjoy!   So  besides this DL lecture  I once again  worked on this robotics project.   And  I have to say  working on it this semester just wasnt necessary It was really not that interesting anymore  and I really just wanted to learn new stuff.   But I was still hoping for a paper  which in the end was never successfully published :(   Now  my second project course this semester  on the other hand  was again about RL but was amazing!   I had to thoroughly read a paper  and actually reimplement it and reproduce its results! Which was a lot of fun!   I often say it  and Ill say it again. Reimplementing a paper and recreating its results is one of my favorite projects to recommend!   I even wrote a blog post about it and submitted it to a top ML conferences blog post track!   But I didnt really know how the process worked back then haha   I did get my reviews  but never received an email telling me that they were released.   So when I randomly checked  I saw the reviews and that I never responded to them haha   Thus  the article was rejected from the ICLR blog post track.   Nevertheless  the project taught me a lot and  at this point  I was pretty confident I wanted to become a top ML researcher!   This goal meant that I needed to strive for the best companies!   My job  at that time  as a student researcher was not completely plateauing  but also not the best anymore.   We started doing research on graph neural networks  but for over a year now  we were still stuck with a lot of the same boring data and feature engineering.   I effectively didnt really learn anything new.   Thats why I wanted to find a new job and not make the same mistake as before  where I stayed for 3 years at the same job   So I applied to dozens of the top ML internships! And I actually got invited to an interview for an applied science internship at Amazon!   That was my first real tech interview! It was really exciting!   Except that I failed miserably.   The more frustrating part was  that the questions were really not that hard it was a rapid-fire  basic ML questions interview.   They were literally asking about the content of the first ML course I mentioned before. The one I completed not even a year ago   But  well  life goes on and I got another interview at a cool startup called Nuro!   This time  it was for an ML engineering internship  and the first interview round was a coding interview!   Again  something completely new to me!   I prepared using Leetcode  but when I saw a blank coding canvas and no preexisting code where I just had to fill in an algorithm  I was so scared.   I failed miserably. Again.   Well  the applications werent going so well. I simply didnt get many more interviews.   So I changed my approach! I directly reached out to a Google DeepMind researcher I found interesting and asked for an internship.   And he got back to me!!! We had an interview call and I felt it went decently well!   But I got rejected   I was done looking for internships  and focused on finding a new job as a student researcher where I could also do my master's thesis!   I decided I had enough of RL and found CV really cool!   But then I thought   how cool would it be if you could talk to an AI about images or even videos!   Thats when I decided multimodal learning was really cool!   But  at my university  there was a problem.   There were no professors working on multimodal learning  and pretty much all of the professors were how do I say it a bit more old school and not thaaaaat much into the new stuff.   There definitely were one or two  dont get me wrong  but they werent into something even similar to multimodal learning.   So I looked outside of my uni  TU Berlin. I wanted to look for a professor who was a bit more active and ambitious.   I read multimodal learning papers and looked at the authors.   I then googled them to see if they could be an option as an advisor for my research and thesis.   And then I found the perfect professor!!!   He was young  and was just about to start as a professor  and before that  he was a postdoc at UC Berkeley and a researcher at Meta!   And he worked on multimodal learning! He was everything I was looking for!   Long story short  I am so happy to have gotten the job and started to work with him  later in my final year.   I still had my goal of getting to big tech  but there are these nice sayings.   Rome wasnt built in a day.   and   All roads lead to Rome.   I.e. Everything takes time  and there are multiple ways to get where you want to get!   So  all in all  this semester  besides this career hassle  I just did a lot of coding!   At my job  for the robotics project  and for this RL paper reimplementation!   But this was still just the first half of my 5th year!!!   My second half was not that eventful haha   Since I failed all my applications for summer internships  I was still doing my best to learn stuff at my  at the time  current job  otherwise  not much interesting stuff happening there.   And at uni  I really focused on my CV!   I took a course on Automatic image analysis and another seminar course on DL for CV  where I had to read several papers on self-supervised learning and present them to the group.   That was so much fun! I just really love reading cool papers :)   I even made my presentation into a mini-series on representation learning  haha   But besides those two courses  I took my second general deep learning course!   This one was finally a bit more advanced!   I learned about things like Representation learning  self-supervised learning  Transformers  GANs  Diffusion models  Graph neural networks  and even ordinary neural differential equations!   And finally  I also did another CV project course  where I wrote a paper/ technical report on!   So there was way more theoretical content this semester  but still a practical project!   Now  you might have noticed that this semester usually should have been my final semester.   Usually  the masters would end after 2 years  but I had actively decided to give myself one more year  mainly to have one semester for an internship and one more for my thesis!   So  this semester was my last one with courses and (since I didnt get an internship) I had one more entire year to focus on doing research with my new professor and then completing my master's thesis.   And that is what I did in my final year!   Year 6I was finally done with uni! At least  it felt like that because I had no more exams.   I started working as a student researcher with this cool professor and started doing research on multimodal learning  specifically  video-moment retrieval.   I read a lot of papers  developed a model that achieved new SoTA performance on the benchmarks I evaluated on  and wrote a paper on it in a very short time!   I even submitted the paper to a top conference. Im telling you  those were some stressful weeks But it still recently got rejected.   And to be honest  I probably understand why.   I rushed it because we chose a deadline that was simply way too close.   I should have taken more time and just submitted it to a later conference so that the paper was overall more solid.   I share more of my learnings and experiences in my Substack newsletter  e.g. the lessons I learned from this rejection and more content that is based on what Im up to as an ML researcher that wouldnt really work here on Medium :)   Now  although it was annoying  I will continue to improve this work and soon submit it to another conference!   Then I remembered that I am still in my final year! I still need to actually complete my degree  lol   Thats why I am currently still in the process of finishing writing my thesis and handing it in.   But  since this is my final year  I also had to think of what comes next!   I thought to myself  either I skip the PhD and become a researcher at a top lab  or I do my PhD.   I mean  how likely was it to skip the PhD?   The cool thing was I already had an offer from my professor for the PhD position  and I was very happy to accept it.   Nevertheless  I still wanted to try out applying to two companies as a research scientist.   One was DeepMind  and  although I thought my chances were  in fact  decent  because I had exactly the combination of different experiences that they were looking for  I got rejected.   But besides DeepMind  I applied for another really cool AI startup. My favorite one  to be precise.   I knew I wouldnt even get invited to an interview. But one evening  I was like  Why not They wont invite me anyway.   But you probably already know where I am going with this. They did invite me!   And I was shocked!!!   The application process was quite tough  and I wanted to really give it my all and see if I am good enough for them.   And  well  long story short  I did get an offer and will work for them starting in a few months.   Once I start my work  I will announce which company it is  dont worry!   I just want to make it cool because  for me  it is a big thing :)   But yeah  anyway  throughout all these years  there was a lot of struggling but also some occasional successes!   I quickly learned that the important thing is to keep moving.   Some people get to where I am now in less time and some in more. But that doesnt matter!   What matters is that you try to improve every day by 1%  overall enjoy what you do  and that you are proud of what you do!   Nevertheless  there are many mistakes you can avoid and not waste any time on if you simply know what they are.   Thats why you might want to read this blog post next.   I there share 7 common mistakes beginner ML students make every year!   7 Mistakes Beginner ML Students Make Every YearDont study LLMs! Youre making a mistake!pub.towardsai.net   Happy learning and ba-bye! U+1F44B


RAG Architecture: Advanced RAG

Since the writing of my last article  not much time has passed  but progress doesnt stand still  and several important changes have occurred. Here  I wont cover the basics  read the original article for that.   The first significant change is the substantial increase in the context window size and the decrease in token costs. For example  the context window size of the largest model Claude from Anthropic is over 200 000 tokens  and according to the latest news  Geminis context window can reach up to 10 million tokens. Under these conditions  RAG (Retrieval-Augmented Generation) may not be required for many tasks (or at least not all of its components) since all the data can fit into the context window. Weve already encountered several financial and analytical projects where the task was completely solved without using a vector database as an intermediate storage. The trend of token cost reduction and context window size increase is likely to continue  reducing the relevance of using external mechanisms for LLMs. However  they are still required for now.   If  however  the context size is still insufficient  different methods of summarization and context compression have been devised. LangChain has introduced a class aimed at this: ConversationSummaryMemory.   llm = OpenAI(temperature=0) conversation_with_summary = ConversationChain( llm=llm  memory=ConversationSummaryMemory(llm=OpenAI())  verbose=True ) conversation_with_summary.predict(input=Hi  what's up?)Knowledge GraphsAs the amount of data LLMs have to navigate grows  the ability to navigate this data becomes increasingly important. Sometimes  without being able to analyze the data structure and other attributes  its impossible to use them effectively. For example  suppose the data source is a companys wiki. The wiki has a page with the companys phone number  but this isnt explicitly indicated anywhere. So  how does the LLM understand that this is the companys phone number? It doesnt  which is why standard RAG wont provide any information about the companys phone number (as it sees no connection). How does a person understand that this is the companys phone number in this case? Its simple  the page is stored in a subdirectory called Company Information. So  a person can understand what the data means from the convention of how the data is stored (i.e.  from the structure or metadata) and use it effectively. For LLMs  this problem is solved with Knowledge Graphs with metadata (also known as Knowledge Maps)  which means the LLM has not only the raw data but also information about the storage structure and the connections between different data entities. This approach is also known as Graph Retrieval-Augmented Generation (GraphRAG).   Graphs are excellent for representing and storing heterogeneous and interconnected information in a structured form  easily capturing complex relationships and attributes among different types of data  which vector databases struggle with.   Example of a Knowledge GraphHow to create a Knowledge Graph? This is an interesting question. Usually  this process involves collecting and structuring data  requiring a deep understanding of both the subject area and graph modeling. This process can largely be automated with LLMs (surprise U+1F642). Thanks to their understanding of language and context  LLMs can automate significant parts of the Knowledge Graph creation process. By analyzing textual data  these models can identify entities  understand their relationships  and suggest how best to represent them in a graph structure.   A vanilla RAG looks something like this:   The modified process will look like this:   So  in fact  this is an ensemble of a vector database and a knowledge graph. As I mentioned in the section on ensembles in the previous article  they generally improve accuracy and often include a search through a regular database or by keywords (e.g.  Elasticsearch).   I wont describe the vector retriever as it is covered in the first article. But lets look at the Knowledge Graph Retriever. As mentioned above  the most obvious way is to ask the LLM. For example  a user asks a question about the companys phone number:   If you do this in code  you can ask to format the found entities in JSON format or use with_structured_output from LangChain. So  the entities from the question are extracted  what next? Next  well look at 100500 use cases from our company on how we applied this U+1F602. Just kidding. Next  we need to search for these entities in the Knowledge Graph. How this is done depends on where the graph is stored.   There are already many graph storage solutions (though companies often make their own versions)  so lets take Nebula as an example.   documents = parse_and_load_data_from_wiki_including_metadata() graph_store = NebulaGraphStore( space_name=Company Wiki  tags=[entity] ) storage_context = StorageContext.from_defaults(graph_store=graph_store) index = KnowledgeGraphIndex.from_documents( documents  max_triplets_per_chunk=2  space_name=space_name  tags=tags=[entity] ) query_engine = index.as_query_engine() response = query_engine.query(Tell me more about our Company)As you can see  the search is not much different from searching in a vector database  except that we search for attributes and related entities  not similar vectors. Returning to the first question  since the wiki structure was transferred to the graph  if everything worked correctly  the companys phone number would be added as a related entity in the graph.   Then  we pass these data and the data from the vector database search to the LLM to generate a complete answer.   It looks simple  but there are a few problems.   Access ControlThe first problem is that access to data may not be uniform. In the same wiki  there may be roles and permissions  and not every user can potentially see all the information. This problem also exists for search in the vector database. So  the issue of access management arises. This problem is further complicated by the fact that there are many different approaches and their hybrids  and for example  anyone who has worked with SharePoint knows that those who have will not laugh at the circus.   There is at least Role-Based Access Control (RBAC)  Attribute-Based Access Control (ABAC)  and Relationship-Based Access Control (ReBAC) and their combinations. Generally speaking  User Directories (like Active Directory)  for example  also represent a graph where the access question is approximately Is there a path from node user U to node resource R. If such a path exists  access is granted.   Permissions and categories are also a form of metadata  and for this whole system to work  these metadata must be preserved at the Data Ingestion stage in the knowledge graph and vector database. Correspondingly  when searching in the vector database  it is necessary to check on the found documents whether the role or other access attributes correspond to what is available to the user. Some (especially commercial corporate vector) databases already have this functionality as standard. This will not work if the data was embedded in the LLM during training U+1F642. Here  one has to rely on the LLMs reasonableness  and I would not do that for now.   Additionally  it is possible to put a censor (guard) on top  filtering the models output if something slips through. Everyone knows Lakera; our company also developed a similar product.   Ingestion and ParsingData needs to be somehow inserted into the graph  as well as into the vector database. However  for the graph  the format is critical  as it reflects the data structure and serves as metadata. Here begins the nightmare of all data scientists  also known as PDF format. You can put everything in a PDF: tables  images  text  graphics. But getting it all back out is sometimes impossible (especially nested tables).   There are different frameworks and libraries that do this with varying degrees of success  LLama Parse being the most notable one.   Unfortunately  there is no good solution for this yet  and sometimes it is easier to use OCR or recognize a document image instead of parsing. Maybe someone will create a separate model focused only on parsing PDFs into something more acceptable  but dreaming doesnt hurt.   In general  the current focus is on improving the quality of answers. Besides using knowledge graphs  there are several approaches:   CRAG (Corrective Retrieval Augmented Generation)Weve seen that RAG sometimes gives incorrect results  and different methods can be used to evaluate them  for example  the LLM itself (or some lighter version). If the results are not relevant  prompt correction  graph search  or even Google search can occur. CRAG goes a bit further  offering a framework that automates this process. Essentially  this is another graph  implementing a state machine (surprise U+1F642)  which looks something like this:   To implement it  its easiest to use LangGraph  which will be discussed further.   Self-RAGSelf-reflective RAG is based on research claiming that this approach provides better results than regular RAG. Overall  the idea is very similar to the previous one (CRAG) but goes further. The idea is to fine-tune the LLM to generate self-reflection tokens in addition to the regular ones. This is very convenient  as there is no need to guess how confident the LLM is and what to do with it. The following tokens are generated:   Retrieve token determines whether D chunks need to be retrieved for a given prompt x. Options: Yes  No  ContinueISREL token determines whether chunk d from D is relevant to the given prompt x. Options: relevant and irrelevantISSUP token determines whether the LLMs response y to chunk d is supported by chunk d. Options: fully supported  partially supported  no supportISUSE token determines whether the LLMs response to each chunk d is a useful answer to the query x. Options represent a usefulness scale from 5 to 1.Using these tokens  a state machine can be built  using the aforementioned LangGraph  which looks something like this:   For more details  see here.   HyDeAnother method  similar to RAG Fusion  is that it modifies the usual RAG retrieval process. HyDe stands for Hypothetical Document Embeddings and is based on the study Precise Zero-Shot Dense Retrieval without Relevance Labels. The idea is very simple  instead of using the users question for searching in the vector database  we use the LLM to generate a response (a virtual hypothetical document) and then use the response for searching in the vector database (to find similar answers).   Why all this? Sometimes users questions are too abstract and require more context  which the LLM can provide  and without which the search in the database makes no sense.   I think this is not an exhaustive review of the new changes; if I forgot something  write in the comments.


Building Visual Questioning Answering System Using Hugging Face Open-Source Models

Visual Question Answering (VQA) is a complex task that combines computer vision and natural language processing to enable systems to answer questions about images.   In this technical blog  we explore the creation of a VQA system using Hugging Faces open-source models. The article begins with an introduction to multimodal models and the VQA task  providing foundational knowledge for understanding how these systems operate.   We then guide you through setting up the working environment and loading the necessary models and processors. By preparing both image and text inputs  we illustrate how to perform visual question answering.   This step-by-step tutorial demonstrates how to leverage Hugging Faces powerful tools to build sophisticated VQA systems  enhancing readers understanding of multimodal AI applications.   Introduction to Multimodal ModelsIntroduction to Visual Questioning Answering TaskSetting Up Working EnvironmentLoading the Model and ProcessorPreparing the Image and TextPerforming Visual Questioning-AnsweringMost insights I share in Medium have previously been shared in my weekly newsletter  To Data & Beyond.   If you want to be up-to-date with the frenetic world of AI while also feeling inspired to take action or  at the very least  to be well-prepared for the future ahead of us  this is for you.   U+1F3DDSubscribe belowU+1F3DD to become an AI leader among your peers and receive content not present in any other platform  including Medium:   To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   1. Introduction to Multimodal ModelsWhen a task requires a model to take more than one type of data  such as an image and a sentence  we call it multimodal. Multimodal models are designed to handle and integrate different forms of input  like text  images  audio  and even video  to perform a variety of tasks.   These models are increasingly important in applications that require a deep understanding of complex data  such as image captioning  visual question answering (VQA)  and multimodal content creation.   One prominent example of a multimodal model is ChatGPT with GPT-4. This model allows users to send text  images  and even audio  making it a versatile tool for a wide range of applications.   GPT-4 can understand and generate human-like text  and when enhanced with multimodal capabilities  it can also interpret images and audio  offering responses that are contextually relevant across different types of data.   Multimodal models have numerous applications across various fields:   Image Captioning: Generating descriptive captions for images by understanding the content within them.Visual Question Answering (VQA): Answering questions about the contents of an image by combining natural language processing with computer vision.Text-to-Image Generation: Creating images based on textual descriptions  useful in creative industries and design.Speech Recognition and Synthesis: Converting speech to text and vice versa  enhancing communication tools and accessibility.Augmented Reality (AR) and Virtual Reality (VR): Integrating multiple data types to create immersive and interactive experiences.In this article  we will explore one of these tasks which is image-text retrieval or matching. In the coming articles of this series  we will cover the rest of these topics.   2. Introduction to Visual Questioning Answering TaskVisual Question Answering (VQA) is a computer vision task involving answering questions about an image. The goal of VQA is to teach machines to understand the contents of images and provide answers in natural language. Questions are typically open-ended and may require an understanding of vision  language  and commonsense knowledge to answer.   VQA has gained attention in the AI community due to its challenge in enabling computers to comprehend image contents similar to humans. It has been suggested that the problem is AI-complete  confronting the Artificial General Intelligence problem. Applications of VQA include aids for visually impaired individuals  education  customer service  and image retrieval.   3. Setting Up Working EnvironmentLets start by setting up the working environments. First  we will download the packages we will use in this article. We will download the Transformers package and the torch package to use Pytorch.   !pip install transformers !pip install torch4. Loading the Model and ProcessorWe will need to load the model and the processor to perform the task. First  to load the model  we need to import the BlipForQuestionAnswering class from the Transformers library.   Then  to load the model  you just need to call the class we imported and use the from_pretrained method to load the checkpoint. We will use the Bleep model from Salesforce for this task  and this is the related checkpoint for this specific task.   from transformers import BlipForQuestionAnswering model = BlipForQuestionAnswering.from_pretrained( ./models/Salesforce/blip-vqa-base)As for the processor  its practically the same. We need to import the AutoProcessor class from Transformers. To load the correct processor  we use the from_pretrained method and pass the related checkpoint. The processors role is to process the image and the text for the model.As for the processor  its practically the same. We need to import the AutoProcessor class from Transformers. To load the correct processor  we use the from_pretrained method and pass the related checkpoint. The processors role is to process the image and the text for the model.   from transformers import AutoProcessor processor = AutoProcessor.from_pretrained( ./models/Salesforce/blip-vqa-base)5. Preparing the Image and TextThe next step is to get the image and the text that we will pass to the processor. The processor will modify the image and the text so the model can understand them.   from PIL import Image image = Image.open(./palestinian_boy.png) imageNow that we have the image  we will check if the model can successfully answer the question and return the answer:   question = how many soldiers are in the picture? 6. Performing Visual Questioning AnsweringFirst  we need to get the inputs that the model can understand. To do that  we need to call the processor and pass a few arguments: the image  the text  and return_tensors set to pt for PyTorch to get a PyTorch tensor at the end.   inputs = processor(image  question  return_tensors=pt) out = model.generate(**inputs) print(processor.decode(out[0]  skip_special_tokens=True))Lets print the inputs to see what it looks like. As you can see  we have a dictionary of multiple arguments: pixel values  input IDs  and the attention mask. Now we have everything.   inputs{pixel_values: tensor([[[[-0.1572  -0.1426  -0.1572    -1.1791  -1.5587  -1.6025]  [-0.1572  -0.1572  -0.1718    -1.1207  -1.5295  -1.6025]  [-0.1864  -0.1718  -0.1864    -1.1207  -1.5149  -1.5879]    [ 0.2807  0.2515  0.2223    0.3975  0.2661  -0.6682]  [ 0.2661  0.2515  0.1931    0.4413  0.2807  -0.6390]  [ 0.2223  0.2661  0.2369    0.4851  0.3829  -0.5222]]    [[ 0.0338  0.0488  0.0338    -1.1218  -1.4519  -1.5270]  [ 0.0338  0.0338  0.0188    -1.0467  -1.4369  -1.5120]  [ 0.0038  0.0188  0.0038    -1.0317  -1.4219  -1.4970]    [ 0.3640  0.3340  0.3040    0.5591  0.4090  -0.5815]  [ 0.3490  0.3340  0.2740    0.6191  0.4390  -0.5515]  [ 0.3040  0.3490  0.3190    0.6642  0.5291  -0.4614]]    [[ 0.3542  0.3684  0.3542    -0.9825  -1.2243  -1.2954]  [ 0.3542  0.3542  0.3399    -0.9256  -1.1958  -1.2811]  [ 0.3257  0.3399  0.3257    -0.9399  -1.1816  -1.2811]    [ 0.6386  0.6101  0.5817    0.8092  0.6955  -0.2573]  [ 0.6244  0.6101  0.5532    0.8519  0.7097  -0.2289]  [ 0.5817  0.6244  0.5959    0.8945  0.8092  -0.1293]]]])  input_ids: tensor([[ 101  2129  2116  3548  2024  1999  1996  3861  1029  102]])  attention_mask: tensor([[1  1  1  1  1  1  1  1  1  1]])}   Finally  we will decode the generated token IDs into a human-readable string  omitting any special tokens.   out = model.generate(**inputs) print(processor.decode(out[0]  skip_special_tokens=True))8   If you like the article and would like to support me  make sure to:U+1F44F Clap for the story (50 claps) to help this article be featuredSubscribe to To Data & Beyond NewsletterFollow me on MediumU+1F4F0 View more content on my medium profileU+1F514 Follow Me: LinkedIn U+007CYoutube U+007C GitHub U+007C TwitterSubscribe to my newsletter To Data & Beyond to get full and early access to my articles:To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   Are you looking to start a career in data science and AI and do not know how? I offer data science mentoring sessions and long-term career mentoring:Mentoring sessions: https://lnkd.in/dXeg3KPWLong-term mentoring: https://lnkd.in/dtdUYBrM


The Mathematics of Small Things: On Grokking and The Double Descent Phenomenon

The Conundrum  To Overfit or Generalize?So heres the thing when training a model  you are often advised never to overfit. Somehow it makes sense because overfitting is when a models algorithm learns its training data so well that it fails to make accurate predictions on new  unseen data. However  understanding when your model begins to overfit can be useful. A model that overfits also shows the point where the objective function for the models algorithm has been optimized. This can be useful in knowing when to stop training.   Conversely  a model that makes accurate predictions on new  unseen data is said to generalize well. The goal of model development is generalization  not overfitting. However  there is often a tension between optimizing the objective function during training and being able to generalize using the model on new data. The goal is not to overfit. Though overfitting isnt desirable  it can serve as a guide to generalization if understood and leveraged accordingly.   For context  a model is trained on training data  evaluated on a validation set  and then tested on a test dataset. In each instance  an error that measures how well the model predicts accurately is measured  training error and test error  respectively. The difference between these errors is often referred to as the generalization error. When small  it means the model generalized well. When large  the model is said to most likely overfit.   There are numerous books  papers  and techniques written on how to ensure a good fit in a model  how to overcome overfitting and how to enhance generalization. That is not the subject of this article. This article explores two observed phenomena (Grokking and Double Descent) in large models regarding how they overfit and generalize  and some speculations about these types of behavior.   GrokkingImagine you have been trying to learn a language. Lets say you have tried everything you can for the past five years. You are bad at it. You arent learning  not even the fundamentals. Then suddenly  one morning after five years of trying  you wake up and you are speaking the language fluently. This described scenario has been observed in large neural networks and is referred to as Grokking. Grokking in machine learning refers to a model suddenly achieving a deep and thorough understanding of the data it is being trained on. This phenomenon is characterized by a sharp and unexpected improvement in performance after a relatively long period of seemingly stagnated or mediocre results. It is as if the model suddenly gets it!   The interesting thing about this phenomenon is that even though it has been observed  it isnt explainable. We dont know why large models behave this way  as it is contrary to the observed behaviors of neural models explained earlier. Models are often nipped right before they begin to overfit to ensure they can generalize on unseen data. Why would a model generalize far after overfitting on a dataset?   Double DescentDouble Descent refers to another phenomenon observed in the training of deep learning models. It describes the relationship between model complexity and performance in large models. Unlike the traditional U-shaped curve usually observed  Double Descent has an additional descent phase that occurs beyond the point where the model fits the training data perfectly. That is  the model at first performs well on new data  starts to overfit  and then starts performing better than the first time.   Simply put  Double Descent is a phenomenon where models appear to perform better  then worse  and then better again as they get bigger.   Differences between Grokking and Double DescentEven though similar and sometimes referred to as the same phenomenon  Grokking is distinct from Double Descent on the following criteria:   Pattern of Model Improvement: Grokking involves a sudden improvement in model performance after a prolonged period of suboptimal performance. Its more about the learning process within a fixed model structure. Double Descent describes a non-monotonic relationship between model complexity and performance  with an initial increase  a degradation at the interpolation threshold  and then an unexpected improvement as complexity continues to increase.Timing: Grokking happens after extensive training  with the model suddenly improving. Double Descent occurs as model complexity is varied  showing different performance phases depending on complexity.Scope: Grokking focuses on the training process and the models internalization of data patterns. Double Descent focuses on the impact of model complexity on performance  highlighting unexpected behavior beyond the traditional bias-variance tradeoff.Underlying Mechanism: Grokking may be related to the model finally understanding intricate data structures and patterns after extensive training. Double Descent relates to how over-parameterized models can find simpler  more generalizable solutions despite their complexity.Even though these are different phenomena  one thing they both have in common is that they veer off from classical machine learning theory of how a model learns and generalizes. A concept that helps explain how and why models learn the way they do classically is the Manifold Hypothesis.   Manifold HypothesisImagine you have a sheet of paper (a 2-dimensional surface) that you can twist  fold  and crumple. Now  this paper exists in a 3-dimensional space (length  width  and height)  but its true intrinsic dimensionality is still just 2D. When the paper is flat  its easy to see that its just a 2D surface. When you crumple the paper  it might appear more complex and seem to fill more of the 3D space. However  it still fundamentally has only two dimensions. If the paper were crumpled  the paper does not fill the entire 3D space but instead exists on a constrained  lower-dimensional surface within the manifold.   The Manifold Hypothesis is a fundamental concept in machine learning that explains how and why models might learn the way they do. The hypothesis suggests that high-dimensional data (such as images  sounds  or other complex data) lies on a lower-dimensional manifold within the high-dimensional space. For example  most realistic images (faces  objects  etc.) do not randomly occupy the entire high-dimensional space but are instead concentrated in specific regions (the manifold). These regions capture the underlying structure and relationships between the data points.   This hypothesis has important implications for understanding how machine learning models  especially deep learning models  operate and generalize.   If a machine learning model can identify and learn this lower-dimensional manifold  it can more efficiently understand and generalize from the data  as any new realistic combination of the features should exist in that manifold.By focusing on the manifold  the model avoids the noise and irrelevant parts of the high-dimensional space  leading to better performance and generalization.SpeculationsWhat might the Manifold Hypothesis have to do with these two unexplainable phenomena? Below are a few speculations o   More Time Required to Unravel the Manifold for Different Dataset Structures: In Grokking  an over-parameterized model suddenly has a Eureka moment after a long time of training. This phenomenon has mainly been observed with algorithmically generated datasets. The Manifold Hypothesis suggests that real-world data has intricate structures. What if there are degrees of intricacies? What if different data types exhibit different degrees of manifold in a higher dimension space? What if this behavior leads to more complexity in how the model learns the information structure  leading to phenomena like Grokking and Double Descent?Correspondence Principle in AI: In physics  a similar dichotomy exists between classical and quantum physics. Quantum physics is the physics of very small things  where atoms and electrons collide or act accordingly. However  classical physics is straightforward  often deterministic and established. The coexistence of these two subfields in the field of physics has been made possible through a reconciliation  that when quantum numbers are large  the predictions of quantum physics match those of classical physics. This is the Correspondence Principle. Maybe Artificial Intelligence needs a correspondence principle  one that connects the phenomena between how large models behave in relation to statistical laws that govern and predict how traditional smaller models behave.Unexplored Laws for Patterns in Complex Data Structures: Like the laws of large numbers  maybe there are laws yet to be discovered for patterns as they pertain to language  arithmetic  and other complex real-world data structures ingested by large models.Learning theory demarcates easily. Lines are drawn like a linear classifier. However  we step into the real world and there are nuances. It depends  we like to say. Many factors that might seem insignificant in theory determine the outcome  the small significant things we overlooked.   In a world fast approaching where we demand machines to think and act like humans  the small  significant things need to be calculated and accounted for. This is the mathematics of really small things. These phenomena are strange until we discover this  hidden beneath the manifold.


Top Important Computer Vision Papers for the Week from 15/07 to 21/07

Every week  researchers from top research labs  companies  and universities publish exciting breakthroughs in various topics such as diffusion models  vision language models  image editing and generation  video processing and generation  and image recognition.   This article provides a comprehensive overview of the most significant papers published in the Third Week of July 2024  highlighting the latest research and advancements in computer vision.   Whether youre a researcher  practitioner  or enthusiast  this article will provide valuable insights into the state-of-the-art techniques and tools in computer vision.   Table of Contents:Diffusion ModelsVision Language Models (VLMs)Video Understanding & GenerationImage Editing & GenerationImage SegmentationMost insights I share in Medium have previously been shared in my weekly newsletter  To Data & Beyond.   If you want to be up-to-date with the frenetic world of AI while also feeling inspired to take action or  at the very least  to be well-prepared for the future ahead of us  this is for you.   U+1F3DDSubscribe belowU+1F3DD to become an AI leader among your peers and receive content not present in any other platform  including Medium:   To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   1. Diffusion Models1.1. Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted DiffusionCan we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt  as effortlessly as creating an image from a textual description?   In this paper  we present Make-An-Agent  a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to-policy generation.   Guided by behavior embeddings that encode trajectory information  our policy generator synthesizes latent parameter representations  which can then be decoded into policy networks.   Trained on policy network checkpoints and their corresponding trajectories  our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs.   We showcase its efficacy and efficiency on various domains and tasks  including varying objectives  behaviors  and even across different robot manipulators. Beyond simulation  we directly deploy policies generated by Make-An-Agent onto real-world robots on locomotion tasks.   View arXiv pageView PDF1.2. Scaling Diffusion Transformers to 16 Billion ParametersIn this paper  we present DiT-MoE  a sparse version of the diffusion Transformer  that is scalable and competitive with dense networks while exhibiting highly optimized inference.   The DiT-MoE includes two simple designs: shared expert routing and expert-level balance loss  thereby capturing common knowledge and reducing redundancy among the different routed experts. When applied to conditional image generation  a deep analysis of experts' specialization gains some interesting observations:   The expert selection shows a preference for spatial position and denoising time step  while insensitive to different class-conditional informationAs the MoE layers go deeper  the selection of experts gradually shifts from specific spatial position to dispersion and balance.Expert specialization tends to be more concentrated at the early time step and then gradually uniform after half.We attribute it to the diffusion process that first models the low-frequency spatial information and then high-frequency complex information. Based on the above guidance  a series of DiT-MoE experimentally achieves performance on par with dense networks yet requires much less computational load during inference.   More encouragingly  we demonstrate the potential of DiT-MoE with synthesized image data  scaling diffusion model at a 16.5B parameter that attains a new SoTA FID-50K score of 1.80 in 512 times 512 resolution settings.   Project pageView arXiv pageView PDF2. Vision Language Models (VLMs)2.1. Understanding Retrieval Robustness for Retrieval-Augmented Image CaptioningRecent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient  lightweight models with strong domain-transfer capabilities.   While these models demonstrate the success of retrieval augmentation  retrieval models are still far from perfect in practice: the retrieved information can sometimes mislead the model  resulting in incorrect generation and worse performance.   In this paper  we analyze the robustness of a retrieval-augmented captioning model SmallCap. Our analysis shows that the model is sensitive to tokens that appear in the majority of the retrieved captions  and the input attribution shows that those tokens are likely copied into the generated output.   Given these findings  we propose to train the model by sampling retrieved captions from more diverse sets. This decreases the chance that the model learns to copy majority tokens  and improves both in-domain and cross-domain performance.   View arXiv pageView PDF2.2. Goldfish: Vision-Language Understanding of Arbitrarily Long VideosMost current LLM-based models for video understanding can process videos within minutes. However  they struggle with lengthy videos due to challenges such as noise and redundancy  as well as memory and computation constraints. In this paper  we present Goldfish  a methodology tailored for comprehending videos of arbitrary lengths.   We also introduce the TVQA-long benchmark  specifically designed to evaluate models capabilities in understanding long videos with questions in both vision and text content. Goldfish approaches these challenges with an efficient retrieval mechanism that initially gathers the top-k video clips relevant to the instruction before proceeding to provide the desired response.   This design of the retrieval mechanism enables the Goldfish to efficiently process arbitrarily long video sequences  facilitating its application in contexts such as movies or television series. To facilitate the retrieval process  we developed a MiniGPT4-Video that generates detailed descriptions for the video clips.   In addressing the scarcity of benchmarks for long video evaluation  we adapted the TVQA short video benchmark for extended content analysis by aggregating questions from entire episodes  thereby shifting the evaluation from partial to full episode comprehension.   We attained a 41.78% accuracy rate on the TVQA-long benchmark  surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows exceptional performance in short video comprehension  exceeding existing state-of-the-art methods by 3.23%  2.03%  16.5% and 23.59% on the MSVD  MSRVTT  TGIF  and TVQA short video benchmarks  respectively. These results indicate that our models have significant improvements in both long and short-video understanding.   Project pageView arXiv pageView PDF2.3. NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language ModelsCapitalizing on the remarkable advancements in Large Language Models (LLMs)  there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs to generalize navigational reasoning and diverse language understanding.   However  a significant discrepancy in agent performance is observed when integrating LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous downstream specialist models. Furthermore  the inherent capacity of language to interpret and facilitate communication in agent interactions is often underutilized in these integrations. In this work  we strive to bridge the divide between VLN-specialized models and LLM-based navigation paradigms  while maintaining the interpretative prowess of LLMs in generating linguistic navigational reasoning.   By aligning visual content in a frozen LLM  we encompass visual observation comprehension for LLMs and exploit a way to incorporate LLMs and navigation policy networks for effective action predictions and navigational reasoning. We demonstrate the data efficiency of the proposed methods and eliminate the gap between LM-based agents and state-of-the-art VLN specialists.   View arXiv pageView PDF3. Video Understanding & Generation3.1. Video Occupancy ModelsWe introduce a new family of video prediction models designed to support downstream control tasks. We call these models Video Occupancy models (VOCs).   VOCs operate in a compact latent space  thus avoiding the need to make predictions about individual pixels. Unlike prior latent-space world models  VOCs directly predict the discounted distribution of future states in a single step  thus avoiding the need for multistep roll-outs.   We show that both properties are beneficial when building predictive models of video for use in downstream control.   Project pageView arXiv pageView PDF3.2. VD3D: Taming Large Video Diffusion Transformers for 3D Camera ControlModern text-to-video synthesis models demonstrate coherent  photorealistic generation of complex videos from a text description. However  most existing models lack fine-grained control over camera movement  which is critical for downstream applications related to content creation  visual effects  and 3D vision.   Recently  new methods demonstrated the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still  no existing approach enables camera control for new  transformer-based video diffusion models that process spatial and temporal information jointly.   Here  we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plucker coordinates.   The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge  our work is the first to enable camera control for transformer-based video diffusion models.   View arXiv pageView PDF3.3. Towards Understanding Unsafe Video GenerationVideo generation models (VGMs) have demonstrated the capability to synthesize high-quality output. It is important to understand their potential to produce unsafe content  such as violent or terrifying videos. In this work  we provide a comprehensive understanding of unsafe video generation.   First  to confirm the possibility that these models could indeed generate unsafe videos  we choose unsafe content generation prompts collected from 4chan and Lexica  and three open-source SOTA VGMs to generate unsafe videos. After filtering out duplicates and poorly generated content  we created an initial set of 2112 unsafe videos from an original pool of 5607 videos.   Through clustering and thematic coding analysis of these generated videos  we identify 5 unsafe video categories: Distorted/Weird  Terrifying  Pornographic  Violent/Bloody  and Political. With IRB approval  we then recruit online participants to help label the generated videos. Based on the annotations submitted by 403 participants  we identified 937 unsafe videos from the initial video set.   With the labeled information and the corresponding prompts  we created the first dataset of unsafe videos generated by VGMs. We then study possible defense mechanisms to prevent the generation of unsafe videos. Existing defense methods in image generation focus on filtering either input prompt or output results.   We propose a new approach called Latent Variable Defense (LVD)  which works within the models internal sampling process. LVD can achieve 0.90 defense accuracy while reducing time and computing resources by 10x when sampling a large number of unsafe prompts.   View arXiv pageView PDF3.4. Shape of Motion: 4D Reconstruction from a Single VideoMonocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches are limited in that they either depend on templates  are effective only in quasi-static scenes  or fail to model 3D motion explicitly.   In this work  we introduce a method capable of reconstructing generic dynamic scenes  featuring explicit  full-sequence-long 3D motion  from casually captured monocular videos.   We tackle the under-constrained nature of the problem with two key insights:   First  we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE3 motion bases. Each points motion is expressed as a linear combination of these bases  facilitating the soft decomposition of the scene into multiple rigidly-moving groups.Second  we utilize a comprehensive set of data-driven priors  including monocular depth maps and long-range 2D tracks  and devise a method to effectively consolidate these noisy supervisory signals  resulting in a globally consistent representation of the dynamic scene.Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes.   Project PageView arXiv pageView PDF4. Image Editing & Generation4.1. DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity PreservationScore distillation sampling (SDS) has emerged as an effective framework in text-driven 3D editing tasks due to its inherent 3D consistency. However  existing SDS-based 3D editing methods suffer from extensive training time and lead to low-quality results  primarily because these methods deviate from the sampling dynamics of diffusion models.   In this paper  we propose DreamCatalyst  a novel framework that interprets SDS-based editing as a diffusion reverse process. Our objective function considers the sampling dynamics  thereby making the optimization process of DreamCatalyst an approximation of the diffusion reverse process in editing tasks.   DreamCatalyst aims to reduce training time and improve editing quality. DreamCatalyst presents two modes: (1) a faster mode  which edits the NeRF scene in only about 25 minutes  and (2) a high-quality mode  which produces superior results in less than 70 minutes. Specifically  our high-quality mode outperforms current state-of-the-art NeRF editing methods both in terms of speed and quality.   Project pageView arXiv pageView PDF5. Image Segmentation5.1. Ref-AVS: Refer and Segment Objects in Audio-Visual ScenesTraditional reference segmentation tasks have predominantly focused on silent visual scenes  neglecting the integral role of multimodal perception and interaction in human experiences.   In this work  we introduce a novel task called Reference Audio-Visual Segmentation (Ref-AVS)  which seeks to segment objects within the visual domain based on expressions containing multimodal cues.   Such expressions are articulated in natural language forms but are enriched with multimodal cues  including audio and visual descriptions. We construct the first Ref-AVS benchmark to facilitate this research  which provides pixel-level annotations for objects described in corresponding multimodal-cue expressions.   To tackle the Ref-AVS task  we propose a new method that adequately utilizes multimodal cues to offer precise segmentation guidance. Finally  we conduct quantitative and qualitative experiments on three test subsets to compare our approach with existing methods from related tasks.   The results demonstrate the effectiveness of our method  highlighting its capability to precisely segment objects using multimodal-cue expressions.   Project pageView arXiv pageView PDFIf you like the article and would like to support me  make sure to:U+1F44F Clap for the story (50 claps) to help this article be featuredSubscribe to To Data & Beyond NewsletterFollow me on MediumU+1F4F0 View more content on my medium profileU+1F514 Follow Me: LinkedIn U+007CYoutube U+007C GitHub U+007C TwitterSubscribe to my newsletter To Data & Beyond to get full and early access to my articles:To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   Are you looking to start a career in data science and AI and do not know how? I offer data science mentoring sessions and long-term career mentoring:Mentoring sessions: https://lnkd.in/dXeg3KPWLong-term mentoring: https://lnkd.in/dtdUYBrM


RouteLLM: How I Route to The Best Model to Cut API Costs

large language models have shown amazing capabilities in a variety of tasks  but there is a big difference in their cost and capabilities.   Claude 3 Opus  GPT-4  and others are high in performance  but they are also high in cost. So thats why were making deals now. The trade-off is: use the best  brightest  and most expensive  or go for something cheaper  faster  and less capable.   But what if there was a better way? This leads to the dilemma of deploying LLMs in the real world.   if youre building something to run a business or help with web research  whatever youre doing with these models  routing all your queries to the biggest  most capable model will give you the highest quality responses  but it can be costly.   some of these projects are blowing thousands of dollars because theyre all relying on GPT-4 or whatever   Of course  you can save money by routing queries to smaller models  but the quality of the responses can go down. GPT-3.5 is cheap  but the quality isnt as good  and it fails on harder tasks   Thats where something like Route LLM comes in.   in this video  we will provide an easy-to-understand explanation of Route LLM  what it is  how it works  what its features and even build an actual application.   If you like this topic and you want to support me:   Clap my article 50 times; that will really help me out.U+1F44FFollow me on Medium and subscribe to get my latest articleU+1FAF6Follow me on my YouTube channelMore info on my discordWhat is RouteLLM?RouteLLM is an open-source framework developed by LM.org that aims to reduce operating costs and maintain high-quality responses by distributing queries between different language models through intelligent routing technology. Simply put  RouteLLM can flexibly select an appropriate model to process queries based on the complexity of the query  thereby saving resources.   solved problemUsing high-performance AI to process every query is like consulting a genius professor for simple questions like Whats the weather like today?  unnecessary and expensive.   In contrast  relying on basic AI to process complex queries could be more efficient. RouteLLM optimizes cost and response quality by intelligently matching queries with appropriate AI models.   How RouteLLM worksQuery Analysis: RouteLLM first analyzes the complexity and intent of each query using natural language processing techniques.Win rate prediction model: It uses predictive modeling to determine the likelihood that the advanced AI will provide a significantly better response.Learning from preference data: RouteLLM is trained on historical data  learning from past queries and user feedback to improve its decisions.Dynamic routing: Based on the predictions  the system routes the query to the most appropriate AI model.Continuous Improvement: RouteLLM continuously updates its algorithms to enhance routing accuracy and efficiency as new data is added.Core features of RouteLLMCost-Effectiveness: Leverage cheaper models for simple queries and use expensive  high-performance models only when necessary.Efficient routing: By using preference data to train the router  it learns the strengths and weaknesses of different models in processing different queries.Data augmentation: Data augmentation technology is used to improve the model's routing performance  including golden-label datasets and LLM-judge-labeled datasets.Advantages of RouteLLMRouteLLM performs well on multiple benchmarks. For example  using GPT-4 Turbo as the strong model and Mixtral 8x7B as the weak model  RouteLLM saves 75% in cost compared to random routing while maintaining high performance.   How do you set up and use RouteLLM?1. Cloning the GitHub Repository:git clone https://github.com/lm-sys/RouteLLM.gitgit clone is a Git command used to create a copy of a specific repository from GitHub (or another Git-based repository hosting service).   2. Navigating to the Cloned Directory:cd RouteLLMTo use RouteLLM  you first need to install it with the command:   pip install routellm[serve eval]Basic Configuration   import os from routellm.controller import Controller os.environ[OPENAI_API_KEY] = sk-XXXXXX os.environ[ANYSCALE_API_KEY] = esecret_XXXXXX client = Controller( routers=[mf]  strong_model=gpt-4-1106-preview  weak_model=groq/llama3-8b-8192  )Here  mf is the recommended router model. The strong_model specifies the advanced AI (in this case  GPT-4)  and then weak_model specifies a less complex AI (in this case  use groq/llama3).   Router settingsRouteLLM provides a variety of router options  including matrix factorization-based router (mf)  BERT-based classifier (bert)  LLM-based Classifier and Weighted Elo Calculation. You can choose the most suitable router according to your needs:   # Setting Different Routers routers = [ 'mf'  # Matrix Factorization 'sw_ranking'  # Weighted Elo Calculation 'bert'  # BERT Classifier 'causal_llm' # LLM-based Classifier ] # Selecting a Router chosen_router = 'mf'Setting the Thresholdpython -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yamlThis command determines at what level of questions the advanced AI will be asked. Here  it is set to ask the advanced AI for 50% of the total questions.   Usage:response = client.chat.completions.create( model=router-mf-0.11593  messages=[ {role: user  content: Hello!} ] )By doing this  RouteLLM analyzes the question and directs it to the appropriate AI model. As you see in the prompt  we prompt hello and this question can be answered by any model we dont need an expensive model to answer this question   import os from routellm.controller import Controller # Set environment variables for API keys os.environ[OPENAI_API_KEY] = sk-proj-S9UotthZt3QLgrUTouvMT3BlbkFJ3ETijcqlmyL6F1wsX4LU os.environ[GROQ_API_KEY] = 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 # Check if the environment variable is set correctly api_key = os.getenv(OPENAI_API_KEY) print(fOPENAI_API_KEY: {api_key}) # Initialize the Controller client = Controller( routers=[mf]  # List of routers  e.g.  mf for matrix factorization strong_model=gpt-4-1106-preview  # Specify the strong model to use weak_model=groq/llama3-8b-8192 # Specify the weak model to use ) # Selecting a Router chosen_router = 'mf' response = client.chat.completions.create( model=router-mf-0.11593  messages=[ {role: user  content: Hello!} ] ) output_content = response['choices'][0]['message']['content'] print(output_content)Conclusion:RouteLLM is an innovative tool that allows you to use AI technology more economically and efficiently. It seems particularly useful for companies operating large-scale AI services or startups that want to provide high-quality AI services on a limited budget. Whether you are working on daily applications or handling complex AI tasks  RouteLLM is an option worth considering.   U+1F9D9U+2642 I am an AI application expert! If you want to collaborate on a project  drop an inquiry here or Book a 1-On-1 Consulting Call With Me.   Source of informationRouteLLM: An Open-Source Framework for Cost-Effective LLM RoutingRouteLLM PaperGitHub  lm-sys/RouteLLMHow I Build My App in Minutes Using Tasking AI  Open SourceThis video is the ultimate beginners guide to using the brand-new Open-source Tasking AI to build applications and apub.towardsai.net   DeepSeek-coder + llama 3 How I Build Application with One PromptI wanted to know if Maestro could make a video game. So  I asked it to create a game for me  and theres one rule: Imlevelup.gitconnected.com   [Ollama libraries U+1F999] Run Any Chatbot Free Locally On Your ComputerIll show you how to create any chatbot locally for free in just a few minutes. Well make a chatbot that canlevelup.gitconnected.com