Beyond LLMs: Compounds Systems  Agents  and Whole AI Products

A Framework for Building Great AI Products   The other day  I found myself reflecting on a classic concept that I was taught in business school  Maslows hierarchy of needs  a simple but powerful framework for understanding human motivation  with basic physiological needs at the foundation (air  food  water  shelter  sleep  clothing ) and the pursuit of self-actualization at the pinnacle.   This got me thinking  in the world of tech (especially AI) and products  what is an equivalent? I mean  users always have needs  and needs in the product vary significantly subject to the use case and the problem being solved  but a spectrum definitely exists. Is there a model or a framework we can use to identify what constitutes the right product for customers and what customers would expect of the product? Luckily  Geoffrey Moores Crossing the Chasm provides some answers. In his book  Moore references Levitts Whole Product Model  and goes further to simplify by introducing the Simplified Whole Product Model.   In this post  we will internalize Moores model  expand it  and show how it can be applied specifically to AI products (applies to any product as well). Well dive into the trade-offs inherent in building AI applications and illustrate these concepts with real-world examples.   My goal is that after you read this post  you should have a mental model and a framework for building great/usable AI products  which would help you not only think about the technology  but also how it fits in the big picture.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   The Whole Product Primer (Plus its Descendants)The Whole Product model revolves around the idea that a core/generic product must be complemented by additional services and interfaces (aka enablers) making up the Whole Product which should provide a solution to the customers problem and to address their needs.   In Geoffery Moores book  the core/generic product is defined as the fundamental offering or technology that a company produces  which may not be sufficient to fully solve the customers problem or meet their needs.   This is where the outer ring comes into play. It represents the whole (expected) product  which is divided into sectors. This outer ring encompasses all the additional elements that customers expect or require to make the core product fully functional and valuable to them  lets call them the enablers.   The Adapted (Simplified) Whole Product ModelIn the tech industry  companies often prefer to build upon existing open-source projects or technologies rather than developing everything from scratch. These companies focus on adding unique value through layers of customization  support  consulting services  integrations  and proprietary patterns  creating a whole product that is more than the sum of its parts.   Furthermore  any successful technology is bound to become commoditized over time  a strategy we often see in tech employed by competitors who gain from doing so  forcing value into higher layers in the value chain (which they usually have thus wanting to commoditize). Recognizing this  companies need to continually innovate and differentiate their offerings to maintain a competitive edge (related  see a previous post on AI market dynamics and what companies in the space focus their efforts on).   Therefore  lets adapt the simplified whole product model with two key adjustments. First  well shift from fixed sectors to a more modular  petal-like structure. This reflects the interconnected yet distinct components that comprise the whole product layer. Second  well introduce a new layer above the whole product layer  called the differentiated product layer. This layer will highlight the unique value propositions that set companies and their products apart  showcasing how they create the most value for their customers.   To be more concrete  lets show how this can be applied to Slack for example (this is just for illustration purposes  the real differentiators could very well be very different).   In addition to representing the products enablers differently using petal-like modular components  we added a new layer to highlight the differentiators. In the example above and in the case of Slack  enablers could be threads  Slack Connect  the workflow builder  and/or Slack AI.   We are very close to being done here with the adaptations  so we will add one last thing to our new framework. In addition to the differentiated layer  we would like to model customizability for products. I.e.  one customers whole product may not be the same for another. I.e.  not all customers desire exactly the same features  so its important to cater based on customers constraints/needs. For example  generically  some customers value safety/security over cost  others might value speed  etc.   Lets continue the slack example. Slack might have different customers to cater for. Enterprise customers  use it mainly as a means for company-wide communication  in that case  the focus will be security and compliance with the companys communication policy  leading to:   Prioritized Enablers: Enterprise-grade security  granular permissions  compliance features (e.g.  data retention policies)Emphasized Differentiators: Slack Connect for secure external collaboration  integration with enterprise security toolsAnother use-case  focus area might be on developers  and Slack being part of their dev/test workflows. In that case  the focus will be on developer productivity and collaboration  leading to:   Prioritized Enablers: Integrations with development tools (e.g.  GitHub  Jira)  code snippets  powerful searchEmphasized Differentiators: Workflow Builder for automating tasks  Slack AI for code suggestions and knowledge retrievalThe takeaway here is that versatility can be a core differentiator on its own because it allows for tailored product experiences. Another way to look at it is that the constraint being imposed defines the core value proposition of the product and how it is shaped to best serve and differentiate in a particular space.   In our example  Slack can tailor its offering to different customer segments  highlighting the features and capabilities that are most relevant to each group. This customization not only enhances the user experience but also strengthens Slacks value proposition in a competitive market.   Towards Whole AI Products (aka Systems)Hopefully  you have a handle on the adapted simplified whole product framework by now. Next  we will focus on using the framework and mapping it to the super exciting world of AI applications.   Key Ingredients to Building AI ApplicationsBefore the mapping  lets do a quick primer on the core ingredients of AI products and applications (a sample not an exhaustive list). We will cover the key ideas  but we wont delve into the technical intricacies. For that  there are many resources available  some of which I will be referencing as we go for further reading.   LLMs AND/OR SLMsIn a previous post  I introduced the model product possibilities frontier  a framework for studying the tradeoffs and use cases of large language models (LLMs) and Small Language Models (SLMs)  which I will not be repeating here for brevity. That said  the choice of which models and their size to use is a key ingredient for building generative AI applications and products.   Here are a few considerations/questions to ask yourself when reasoning about the tradeoffs:   What are your most favorable constraints? Is it speed  quality  cost  etc?What about privacy? Do you value data staying in-house (Small models are easier/cheaper to deploy  train  and serve on-premise)How are you going to evaluate the performance of your AI applications that make use of these models?Is a smaller model easier to test and evaluate (think about the specificity as truth vs the versatility of LLMs which introduces more variability/hallucination  and thus makes it harder to test)While we did not call it out explicitly  large or small models can be fine-tuned and aligned. This is covered in greater detail in this post.   Retrieval Augmented Generation (RAG)Id say 2023 was the year of RAG. We went from naive RAG to Advanced RAG. I liked naive tbh  it communicated simplicity  but well  these days advanced is perceived as better  something we are yet to fix  but thats a different story U+1F642. This paper provides more details.   RAG workflows are comprised of many moving pieces and optimizations. The goal is to retrieve the best content to augment the context for LLMs (text generation) with necessary information. In that case  LLMs become curators rather than innovators/generators of sorts (they shape the retrieval results and make them relatable as an output to a user but are not the source of knowledge themselves). To give you an idea of the moving pieces involved with RAG  here is a rough brain dump (feel free to surf the mindmap as you please  I will not enumerate the details here for brevity).   When considering RAG for building AI applications  some questions come to mind around tradeoffs and decisions  usually between RAG  long context  and Fine-tuning. Again  we wont cover details  but here are a set of questions that you can ask to inform your decision.   Does the application require access to external data sources to provide accurate and up-to-date responses (RAG usually makes sense if data freshness is important  especially since language models are point-in-time trained)?Is it crucial for the model to adapt its behavior  writing style  or domain-specific knowledge to match specific requirements (RAG does not customize behavior  fine-tuning would make sense if behavior customization is a goal)?How critical is it to minimize the risk of the model generating false or fabricated information (hallucinations)?How much labeled training data is available for fine-tuning? Does it adequately represent the target domain and tasks?How frequently does the underlying data change? How important is it for the model to have access to the latest information?Is it important to understand the reasoning behind the models responses and trace them back to specific data sources?How important is minimizing computational costs for your project or organization?Do your typical queries require multi-step reasoning (complex queries or simple questions)?How important is the ability to scale your solution to handle a large number of queries?Finally  here is a short guide I created to help you make informed decisions about RAG/Fine-tuning if you wish to use it:   For more information  check the below papers  which I found very useful in understanding the differences and the trade-offs:   [2407.16833] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach[2401.08406] RAG vs Fine-tuning: Pipelines  Tradeoffs  and a Case Study on Agriculture[2312.05934] Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsRAG today has become synonymous with building AI applications in some contexts. Whats clear is that RAG is not one component  its a system comprised of many moving pieces with levers to turn on/off for what makes sense most subject to context and use-case.   Agents ft. Agentic OR Agentless!In addition to the model (LLM/SLM)  RAG  there is the notion of agents and agentic workflows (also agentless to counter U+1F642). While this is again not going to be a deep-dive  lets cover the basics.   What are agents  what is agentic behavior  and why agentless sometimes?   The notion of agents is not new. Agents have existed for decades (see this for examples)  they are officially called Intelligent agents. Below is the definition of an Intelligent Agent.   In intelligence and artificial intelligence  an intelligent agent (IA) is an agent acting in an intelligent manner. It perceives its environment  takes actions autonomously in order to achieve goals  and may improve its performance with learning or acquiring knowledge. An intelligent agent may be simple or complex: A thermostat or other control system is considered an example of an intelligent agent  as is a human being  as is any system that meets the definition  such as a firm  a state  or a biome.[1]   Whats changed is that with the advent of LLMs is that agents got a capability boost  from symbolic  rule-based  predefined simple actions with low autonomy (see the history post for more details  you may be reminded of expert systems) to being able to understand and generate natural language  learn and adapt across diverse domains  and perform complex  autonomous actions. In todays context  An agent is a software entity possessing autonomy  goal-oriented behavior allowing it to operate and generalize cross-domains and take complex actions.   Agentic behavior  in this context  refers to an agents ability to operate independently  make decisions aligned with its objectives  and execute actions (potentially with tools/functions-calling ) to achieve those goals. The level of agency can vary based on factors like the complexity of the environment  the agents goals  and the degree of user supervision required. More agentic systems can operate autonomously in intricate environments  pursue complex objectives  and utilize advanced techniques such as planning and tool use   Finally  there is the notion of flow-engineered / AGENTLESS which relies on determinism and only interfaces with language models for specific clarifying actions  in a sense similar to intelligent agents of the past  with the exception of having access to external intelligence capable of better identifying areas where the predefined action could be taken.   To simplify your life  Ive included this visual below (higher resolution here) to help you build a clearer mental picture of agents/agentic.   Other componentsBesides agents  RAG  the models  there are multiple other ingredients that go into building an AI applications  going through each and every one is out of scope for this post  but here is a non-exhaustive list for reference:   Data Pipeline: System for collecting and processing data  think extractions  transformation.Knowledge Base: where the processed knowledge/data is stored.User Interface: Web or app interface for users.Query/prompt Cache: avoid unnecessary query round-trips which can greatly reduce costs.APIs: To interface with other systems.Infrastructure: an important component that is usually overlooked  where to host the model/app  how to scale it  etc.Observability: be able to log  monitor  trace an AI application.Model Gateways: to interface between the user-query and its destination. Along the way  it makes sure the query is authenticated/authorized  masked/audited for sensitive content (e.g.  PII)  and finally routed to the best model to serve the query (best here is dependent on the use-case  see this post)<Many more>As I was writing this  I came across this blog post  which discusses the technical details of some of the most used components for AI applications.   Compounds AI SystemsYou have come a long way brave reader  the end is near  and you shall be rewarded. So far we have been separately covering important components and ingredients that are key to the making of AI applications  but what makes the interconnection of these components towards achieving a shared goal? A system!   A system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole   Zaharia et. al recently introduced the notion of Compound AI Systems. In their post  they define it as:   A system that tackles AI tasks using multiple interacting components  including multiple calls to models  retrievers  or external tools. In contrast  an AI Model is simply a statistical model  e.g.  a Transformer that predicts the next token in text.   The authors also emphasize the complexity of designing AI systems:   While compound AI systems can offer clear benefits  the art of designing  optimizing  and operating them is still emerging. On the surface  an AI system is a combination of traditional software and AI models  but there are many interesting design questions. For example  should the overall control logic be written in traditional code (e.g.  Python code that calls an LLM)  or should it be driven by an AI model (e.g. LLM agents that call external tools)? Likewise  in a compound system  where should a developer invest resources  for example  in a RAG pipeline  is it better to spend more FLOPS on the retriever or the LLM  or even to call an LLM multiple times.   In their post  they showcase a table of AI systems  and the components they are composed of. Additionally  they highlight the need for optimization across the chosen components to build reliable AI systems. Below we extract the components mentioned in the post and categorize them into Ops (i.e.  operations)  Tools  Context/Knowledge  and models.   If you remember in the previous section  we covered similar components and more as ingredients to build AI applications. The takeaway here is that building reliable AI applications takes a system not a singleton component. I.e.  the whole is more than the sum of the parts   Another way to visualize it is to consider a dashboard looking like a cockpit with all knobs needed to build your AI application  here is an example of what that could look like:   Without abstraction  youd have to configure all these knobs manually (i.e.  youd have to understand what each of these means). Nowadays  there exist many frameworks to do the orchestration which to a good extent abstracts away some if not all these details. Is that a good thing? I will let you decide. My take? It can be a good thing if you are experimenting  learning  but if reliability  performance  and security are concerns (and they should be)  youd still have to understand what all these knobs mean before you pick up automation/orchestration tooling. Think of it this way  do pilots just take on their license without understanding what each and every knob in their cockpit means? I would guess not! But when they do  they can auto-pilot if they choose to because at any point they CAN switch back to pilot-mode and turn on the right knobs to fly the plane safely.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   From Compound Systems to Whole AI ProductsNow that we understand the key ingredients needed to build AI applications and compound AI systems  which is the technical pattern we will use to describe the components of an AI application and how they intermingle  lets go ahead and map that back to our adapted simplified whole product framework.   Note: While having a technical system encapsulating the main function(s) of the product is great  shipping and building whole products take more time/effort (to execute) than the technical parts.   As you can see in the diagram above (higher resolution here)  we took the components of compound AI as we categorized them in the previous section  and mapped them to the generic/core (right in the middle)  and the whole product layer comprised of one or more enablers. You may notice that we left out the differentiated product layer  thats intentional. We will cover that in a coming section. What about the constraints? Lets model them as well well.   The constraints will heavily depend on the use-case  I used Enterprise here as an example. For enterprise AI use-cases  safety  and reliability are important concerns. Using the constraints  we put emphasis on specific parts of the whole product  highlighting key enablers. In that case we chose legal  ops  gateway  and UX.   Different use-cases will place different emphasis on the whole product  resulting in some layers being more important than others. Some use-cases even simplify the whole product by losing unneeded enablers  making the whole product leaner and more directed towards solving the problem/use case at hand.   Defensibility AND Compound MOATSPreviously we took a tour to compare and contrast the current AI Market Landscape. We showed how companies that have a mission to better something other than just the model might have better odds in surviving in a competitive market (I.e.  AI as an enabler vs. AI as the core product). We have also shown how companies are releasing open-source language models  which increases competitiveness and commoditizes the model layer completely making it pertinent for startups and companies to see defensibility through differentiation  i.e.  what is the companys MOAT?   For defensibility  lets summarize the most prominent strategies:   Having strong using communities and strong user engagement.Transitioning from Foundational Models to Purpose-Based ApproachesBuilding Layers of Value Beyond the ModelDifferentiating at Various Layers of the AI StackLets briefly get into each.   Fostering a strong community and high user engagement: This involves cultivating a rapidly growing user base  harnessing the power of network effects  and creating a vibrant community that engages users across different generations. I.e.  Who will use my product  what value to I provide beyond just the model  and why do I have a community in the first place?Transitioning from general foundational models to purpose-built applications: By focusing on specific user needs and problems  companies can tailor their AI solutions to provide more value and differentiate themselves in the market using existing business models  E.g.  I already a social network  I make good money from Ads  how can I add more value to the existing community by incorporating AI?Building layers of value beyond the model: Invest in research to continually improve models and applications  leverage proprietary data (data as moat) for enhanced performance (after all garbage in  garbage out  gold in  gold out)  and continuously refine products based on user feedback. By building a loyal customer base and offering unique value propositions  companies can establish a strong competitive advantage.Differentiate by focusing various layers of the AI stack: This can involve developing superior AI models or smaller niche models (focusing on a tiny use-case but beating anyone else at doing it)  providing scalable and efficient AI infrastructure  or creating user-friendly interfaces and seamless integrations (a GPT store  for example?). Each layer presents an opportunity for differentiation and can contribute to a companys overall defensibility.These are just but some strategies that can be used to build moats  it is rarely a single component  its the sum of multiple to make a better whole defensible product. Compound MOATs  are the way! The last strategy is the one with lowest chances of surviving alone  so Id consider at least two of the above strategies to start differentiating. Some questions to ask:   What processes do you have in place to ensure that AI models are being leveraged as enablers rather than being treated as end products?What strategies are you employing to rapidly grow your user base  create network effects  and foster a sense of community?What investments are you making in research  data  product refinements  and customer acquisition to build layers of value?What resources are you allocating to differentiate your company at the model layer  infrastructure layer  or application layerHow are you evaluating and prioritizing potential areas of differentiation to ensure a sustainable competitive advantage?Adding The Differentiated Product LayerAlright alright alright  now that we understand moats/defensibility strategies  how do we model them back into our framework?! Using any (or additional) defensibility strategies to differentiate  additional components are added to the differentiated product layer in the model. In that case  we added strong community  integration with partners  a store/marketplace  innovations at the application layer (value above the model)  and unique data. This layers makes a companys set of Compound MOATs  which are also what create brand differentiation  loyalty  retention  etc.   AI Whole Products in PracticeIts 2024  almost two years after the release of ChatGPT  almost 70 years after the perceptron  the first manifestation of neural networks (see this post for more details)  and ~40 years after the creation of expert systems which was the closest Applied AI could get. In the post  I go into the details of why expert systems did not pan out (and partially led to an AI winter)  but for brevity  it was a consumption gap  what we had back then in terms of compute  community  and technology was a far cry from where we are today.   With LLMs showing a glimpse of what can be achieved with natural language  and with the maturity of predictive AI and deep neural networks  applied AI is a reality now more than ever. In this section  we show hope AI applications are built using compound AI systems in the wild. There are many sources of knowledge about applications of AI that can be found on the internet. I chose to use the Federal AI use-case inventory to extract some examples use-cases  followed by a real case of how Uber and OpenAI make use of compound AI systems to build whole AI products and map them to our adapted simplified whole product framework.   Federal AI Use-Cases ExamplesBelow is the breakdown for 6 example use-cases from the inventory after we have applied the framework (use the codes to find them in the inventory).   Note: Higher resolution of the image below can be found here.   Example 1: TowerScout (HHS-00222023)Problem: Identifying potential sources of Legionnaires Disease outbreaks during investigations.   Constraints: Accuracy  speed of detection  ability to process aerial imagery.   Core Product: Object detection and image classification models trained to recognize cooling towers.   Enablers:   Data Pipeline: System to acquire  process  and store aerial imagery.Knowledge Base: Geographic data on building locations  potential water sources.Tools: Image annotation tools  model training infrastructure  visualization software (GIS).Differentiated Product Layer:   Integration: Direct integration with CDC outbreak investigation workflows and databases.Unique Data: Access to CDCs epidemiological data for model training and validation.Example 2: USDA Cropland Data Layer (USDA-00262023)Problem: Classifying crop types and land use for agricultural monitoring and statistics.   Constraints: Accuracy  national coverage  consistency over time  ability to handle satellite data.   Core Product: Machine learning algorithms (likely Random Forest) trained to classify crops from satellite imagery.   Enablers:   Data Pipeline: System to acquire  process  and store multi-temporal satellite imagery.Knowledge Base: Ground truth data from farm surveys  historical crop patterns  weather data.Tools: Image processing software  model training infrastructure  geospatial analysis tools.Differentiated Product Layer:   Long-Term Data: Historical CDL data provides valuable insights into agricultural trends.Public Availability: Open access to CDL data makes it widely used by researchers and policymakers.Example 3: Human Resource Apprentice (OPM-00002023)Problem: Time-consuming and potentially subjective evaluation of applicant qualifications in government hiring.Constraints: Accuracy  fairness  ability to process applicant resumes and job descriptions  explainability.Core Product: AI model (NLP and potentially ranking algorithms) trained on data from previous hiring decisions.Enablers:   Data Pipeline: System to acquire and process applicant data from applications and resumes.Knowledge Base: Job descriptions  qualification requirements  competency frameworks.Tools: NLP libraries  model training infrastructure  user interface for HR specialists.Differentiated Product Layer:   Bias Mitigation: Robust testing and evaluation for fairness and adverse impact mitigation.Explainability: Ability for the system to provide clear rationale for applicant rankings.Example 4: HaMLET (Harnessing Machine Learning to Eliminate Tuberculosis)  HHS-00232023 (CDC)Problem: Improving the accuracy and efficiency of overseas health screenings for immigrants and refugees  specifically for tuberculosis.   Constraints: Accuracy  speed (high throughput)  ability to process chest x-rays  potential resource limitations in overseas settings.   Core Product: Computer vision models trained to detect TB from chest x-rays.   Enablers:   Data Pipeline: System for acquiring  digitizing  and storing chest x-rays.Knowledge Base: Large  labeled dataset of chest x-rays with confirmed TB diagnoses.Tools: Image annotation tools  model training infrastructure  potentially lightweight deployment for use on less powerful devices.Differentiated Product Layer:   Public Health Impact: Potential to significantly reduce TB transmission and improve global health outcomes.Resource Efficiency: Automating screening can reduce the need for specialized personnel  making it more feasible in resource-constrained settings.Example 5: RelativityOne (DHS-00262023  Dept. of Homeland Security)Problem: Inefficient and time-consuming document review in litigation  FOIA requests  and other legal processes involving large volumes of documents.   Constraints: Accuracy  speed  ability to handle diverse document formats  legal and ethical considerations around data privacy and access.   Core Product: A document review platform using machine learning techniques (continuous active learning  clustering).   Enablers:   Data Pipeline: System for ingesting  processing  and indexing large volumes of documents.Knowledge Base: Legal frameworks  case law  and other relevant information for model training.Tools: Text extraction and analysis tools  user interface for legal professionals to review and manage documents and results.Differentiated Product Layer:   Enhanced Efficiency: Significantly reduces the time and resources required for document review.Improved Accuracy: ML models can identify relevant documents and patterns that humans might miss.Compliance and Security: Strong focus on data security and compliance with legal and ethical requirements.Example 6: Cybersecurity Threat Detection (HHS-00152023  ASPR)Problem: Effectively analyzing the massive volume of cybersecurity threat data to identify and respond to real threats.   Constraints: Speed  accuracy  ability to handle diverse data sources  evolving nature of cyber threats.   Core Product: AI and ML models trained to detect anomalies and malicious activity in network traffic and other security data.   Enablers:   Data Pipeline: Real-time data ingestion from various security tools (firewalls  intrusion detection systems  etc.)Knowledge Base: Databases of known threats  attack patterns  and vulnerabilities.Tools: Data visualization and analysis tools  security orchestration and automation platforms for incident response.Differentiated Product Layer:   Proactive Threat Detection: AI models can identify emerging threats and zero-day attacks that traditional rule-based systems might miss.Automated Response: AI can automate incident response actions  such as quarantining infected devices  to contain threats faster.Companies & ProductsBeyond the federal AI use-cases  let us apply the framework to products released out in the open by well-known companies and startups. We will be covering Uber  and OpenAI.   Ubers Michael AngeloRecently  I came across this post and this post covering Ubers journey in developing and refining their AI platform  Michelangelo  over the past 8 years. According to the posts  Michelangelo plays a critical role in powering nearly every aspect of Ubers operations  from core functions like ETA prediction and ride matching to fraud detection and customer support. Additionally  since 2023  Uber has been building various internal generative AI applications and platforms to provide a good foundation for building those applications (see this post on how to build platforms for more details). Here is a distribution of their generative AI use-cases/goals:   With that in mind  lets apply our adapted whole product framework to Ubers internal AI use-case with Michaelangelo and building an AI platform.   Problem: Lack of a standardized and scalable system for developing  deploying  and managing ML across Ubers diverse business needs with tiering/prioritization.   Goal: Harness the power of both traditional ML and LLMs to improve core operations (ETA  pricing)  enhance user experiences (customer support  app features)  and boost internal productivity.   Constraints:   Scale: Managing massive data volume and real-time prediction demands of a global user base.Latency: Delivering low-latency predictions for time-sensitive applications.Security & Privacy: Protecting user data  particularly PII  especially when using external LLMs.Collaboration: Supporting efficient workflows for diverse teams of data scientists  ML engineers  and application developers.Adaptability: Rapidly evolving to integrate new AI/ML technologies and adapt to the changing landscape.Cost-Effectiveness: Managing the computational expenses of large-scale AI  optimizing where possible.Core Product: Fine-tuned / Custom self-hosted LLMs tailored for Ubers internal use-cases.Enablers:   Data Pipeline: System for collecting and processing data  think extractions  transformation.Palette: Feature store for managing  sharing  and accessing features across Uber.Data Processing & Prep: Tools for collecting  cleaning  and transforming data for both traditional ML and LLMs.Knowledge Integration: Connecting LLMs to knowledge bases  APIs  and Uber-specific data sources for grounding and context.Tools (part of enablers):   Development: Michelangelo Studio (MA Studio) for UI-based workflows; Canvas for code-driven development  version control  and CI/CD.Training: Horovod  Ray  Spark  support for TensorFlow and PyTorch; specialized tools for LLM fine-tuning and optimization.Serving: Triton Inference Server  Michelangelos real-time prediction service (OPS).Monitoring: Model Excellence Score (MES) for quality assessment  feature monitoring  SLA integration  and LLM performance tracking.Gateways: Ubers Specialized Gateways such as (GenAI  CO Inference) abstracting complexities and providing easier access to AI capabilities.User Interfaces: Michelangelo Studio: Unified UI for managing ML workflows.Legal & Operations (part of enablers):   Security & Compliance: PII redaction  access controls  bias detection  and mechanisms for ensuring responsible AI usage.Cost Management: Tracking LLM usage  setting budgets  and implementing cost optimization strategies.Model Versioning & Artifact Management: Ensuring reproducibility  tracking experiments  and managing model deployments.Differentiated Product Layer:   Scale and Operational Efficiency: Michelangelo and its integrated gateways are built to handle the complexities of AI/ML at Ubers global scale.Internal Platform Expertise: Ubers AI platform team has deep knowledge of the companys unique data  business needs  and engineering environment.Focus on Developer Experience: Tools like MA Studio and Canvas  combined with the abstraction layers of gateways  prioritize developer productivity and ease of use.Hybrid Approach: Combining traditional ML and LLMs through a unified architecture allows Uber to address a wider range of use cases.If you have noticed  and in the mapping we have done so far for Michael Angelo  the whole product is the platform. Its what enables developers to build products that customers love  take their mobile application for example. I have discussed platforms as products or products of the platforms in more length in this post. Feel free to take a refresher trip if you are looking for more details on the distinction.   OpenAIs ChatGPTBy now you most likely have used a variant of ChatGPT  what you have not seen is whats running under the hood to allow you to use the interface exposed and get the chat experience you get. Below is a diagram from an OpenAI talk about what the platform looks like under the hood and what it takes to run ChatGPT and expose to the world.   To get more visibility  lets apply the adapted whole product framework to ChatGPT :   Problem: How to providing accessible  versatile  and powerful AI assistance for a wide range of tasks and queries.   Constraints:   Safety and ethical considerationsScalability to handle massive user demandAccuracy and reliability of outputsCost-effectiveness of compute resourcesCore Product: Large Language Models (GPT series)   Enablers:   Context/Knowledge: Fine-tuning datasets for specific tasks and safety alignmentTool-use: ChatGPT  DALL-E  and Codex for code generation and understandingUX: the ChatGPT web interface + the Mobile appOps (part of enablers):   Scalable infrastructure for model training and inferenceMonitoring and logging systemsUser feedback collection and analysisDifferentiated Product Layer:   GPT Store: Marketplace for custom GPTs created by users and organizationsStrong Community and User Engagement: Rapidly growing user base for ChatGPT as well as an active developer community using OpenAI API (in a sense its become the standard)Continuous Model Improvements: Regular updates (e.g.  GPT-3 to GPT-4) and Integration capabilities with other tools and platformsState-of-the-Art Performance: Leading performance in various language tasksUnique Data and Feedback Loop: Massive web-scraped dataset for pre-training  vast amounts of user interaction data for model improvement.Innovation at Application Layer: ChatGPT plugins ecosystem  Realistic Voice with imitation  Assistant API for creating AI agentStrategic Partnerships: Microsoft partnership for exclusive access to GPT models increasing distribution blast radius to all Azure users.Infrastructure: Access to large-scale infrastructure and compute (partially enabled by the Microsoft partnership as well)The (Adapted) Market Development Life CycleSo far we have been traveling across the lands of the adapted simplified whole product framework. Along the way  we have also covered some real examples to demonstrate how the framework is (or can be) used. It wouldnt be a whole product framework adaptation if we didnt adapt it to Moores Market Development Life Cycle model though.   Note: higher resolution of the image below can be found here.   It all starts with a Generic (core) Product  a barebones model appealing to innovators/techies who prioritize core functionality. If you would pick an open-source LLM (maybe fine-tuned to solve a specific problem?) and just put it to the test  that would be an example of a core/generic product (the enabling technology which is at the heart of making a future whole product possible). Innovators here are tinkering with the tech that you seemingly are building your product around (or that you might have built yourself). Questions they might ask here: how does it fair  do we need it  do we have better alternatives  would we require additional support (skill/knowledge?)  do they (your company) have it? youll neeed to make sure you have answers to those questions.   To cross to the Early Adopters and their desire for somewhat practical solutions  your product should find a way to meet the expectations (aka the Expected Product)  for the problem your customer is trying to solve  what are some of the key enablers you made sure to add to create a Minimum Viable Product (MVP)? Here you must have started to target a specific niche  and started to provide enough enablers in the product that it solves 80% of their use-case (they might be willing to help because now they SEE the value of what you are offering). At this stage  relationships and feedback matter.   Now its the moment of truth  to cross the chasm to the early majority. This stage often makes or breaks your product/value prop. You will have to navigate a tradeoff: maintain the speed and innovation that attracted early adopters while at the same time also addressing the reliability/demands to make this product Whole. Make no mistake  the likelihood of others doing the same is high at this stage  but you will need to cross here anyways. Examples of enablers at this stage:   An efficient pipeline for data acquisition  processing  and storage. Think of Ubers Michelangelo platform  with its specialized data management tools like Palette.User-friendly interfaces  efficient model training infrastructure  observability (think compound systems here  and tailor to constraints). Using our Ubers example  think Michelangelo Studio and their AI gateway (AuthN/Z  routing  etc).Knowledge Integration  connecting the AI to relevant knowledge bases (RAG maybe)  well-defined APIs  and domain-specific data sources to enhance its capabilities.Once you do cross  know you have augmented your product just enough to make it whole  welcome to the land of the pragmatists  and congratulations  you have an augmented whole product with well-defined key-enablers that solve the customers problem.   You are not done though! Now you get a chance to tell the world why you are different  ruffle your feathers  and be ready to differentiate  welcome to the Differentiated Product layer. At this stage  youll need to focus on highlighting your unique value proposition and solidify your maots. Examples here could:   Foster an active community around the product (if you have that already  you might be a winner) and encourage user contributions/feedback. Both Slack and OpenAI have cultivated vibrant communities around their products (there are different ways to do that  but thats not the topic of this post maybe more on this later).Collaborate with key partners to expand reach  access valuable resources  and enhance the products capabilities. For example  OpenAIs partnership with Microsoft exemplifies this  granting them access to compute and distribution Leverage unique datasets  if you have a community  you likely also have data unique to your products/services (with consent of course I hope). Develop and customize your models  and refine your core optimizatoions to create a competitive edge. Ubers Michelangelo leverages their vast ride-sharing data and internal expertise to optimize AI for their specific business needs.As you move through the stages  youll notice how the products complexity increases  natural and reflects the evolving needs and expectations of each customer segment/use-case. The visual above hopefully acts as a guide/framework to highlight the importance adapting your AI product strategy accordingly to achieve success in each phase of the lifecycle. Failing to adapt will leave you behind  while successfully listening and continuously building/iterating can give your company and your product a boost into a temporarily blue-ocean (we will talk about that later) where you excel for what you do.   Putting it All Together: Building Whole AI ProductsYou MADE IT! By now  you understand what it takes to build whole AI products! Lets quickly recap.   In this post  we went together on a journey that started from classic business principles like Maslows hierarchy of needs to the world of compound AI systems AND how they map and transform into whole AI products. Weve explored the critical components of successful AI products and applications  adapting Moores Simplified Whole Product Model along the way  and finally fitted our new framework into Moores infamous Model Development Lifecycle framework (again with some adaptations/opinions). Here are some take-aways from our journey:   Its Not Just About the Model: While LLMs and SLMs are powerful (open-source or not)  they are just one ingredient in the recipe for a successful AI product. And yes open source unlocks many potential benefits (out of scope)  but it does NOT mean it rivals whole products!Compound AI Systems make a good pattern/foundation for whole AI products: The true power of AI is unleashed when you combine models  data pipelines  knowledge bases  retrieval mechanisms (like RAG)  agents  user interfaces  and robust infrastructure (and more) into a cohesive system that works well with the defined constraints.Differentiation is key: In a rapidly evolving AI landscape  establishing a moat (see above) is essential for long-term success. Focus on building strong communities  transitioning to purpose-built applications  creating value beyond the model  and differentiating at various layers of the AI stack. Compound MOATs (read above) are the way to go!Constraints Shape Your Product: Clearly define the problem youre solving and the specific constraints of your target audience. These constraints will guide your choices regarding the core product  enablers  and even the differentiators.The Adapted Whole Product Framework Provides a Roadmap: By considering each layer of the framework  the generic/core product  enablers  constraints  and differentiated product layer  you can develop a complete understanding of what constitutes a valuable and defensible AI product.Building AI products is not a one-size-fits-all endeavor. The examples from the Fed-AI use-case inventory  Ubers Michaelangelo  or OpenAIs ChatGPT (some of many examples in the wild) highlight the different approaches and strategies companies/institutions are employing today to build AI products and applications. By focusing on user needs  and continuously innovating/iterating/discovering  you can navigate the uncertainties of the AI landscape and create AI products that truly deliver on their promise.   With all that said and done  now Its Your Turn  friend:   Think about an AI product you are working on or envisioning. Use the adapted simplified whole product framework and the guiding questions posed throughout this post to analyze its strengths  weaknesses  and opportunities for differentiation. Remember  building successful AI products requires building a perspective that goes beyond just the technology itself  remember the whole is greater than the sum of its parts  so make sure how you connect the parts resonates will with your brand  mission  and strategy.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   Thats it! If you want to collaborate  co-write  or chat  reach on LinkedIn. I look forward to hearing from you!   If you like the article and would like to support me  make sure to:U+1F44F Clap awayU+1F449 Follow on MediumU+1F514 Subscribe to The Technomist NewsletterU+1F514 Follow on: LinkedIn U+007C Twitter U+007C GitHub


Why Polars Destroy Pandas in All Possible Ways for Data Scientists?

Pandas needs no introduction  but this article will dive deep into answering the question of why Polars is better than Pandas (even the Author of Pandas agrees).   You might be aware of some basics like memory and speed improvements  but why? How does Polars do their magic to achieve such high speeds and less memory usage?   This article will provide all the reasons why Polars has an advantage over Pandas as well as what it is lacking in comparison (for now).   Lets jump right into it!   Clean APIThere are so many tricks and hacks you can do with Pandas that probably developers themselves are not aware. Daily usage is no different because If I gave you a piece of code in Pandas like this: data.iloc[:  2:] >= 4 and assuming you dont have hyperthymesia  you would not know what this code does. It is known that developers use Google and AI bots to produce code and do not know everything off the top of their heads  but the point here is different.   The functions that the library provides should be straightforward  clear and dedicated to one use.   That is what Polars provides with their excellent documentation  function names  and overall feel of the library stability. Their expressive API is one of the best parts of the library. It provides such a different insight into working with data that going from one framework to another takes a toll on brainpower and shifts the mindset completely.   Speed and memory optimizationThere are multiple reasons for this  and two main ones are Apache Arrow and Rust. Arrow is a language-independent columnar memory format for flat and hierarchical data  organized for efficient analytic operations. Pandas struggles to utilize this efficiently because of the legacy code and data type extensions internal to the library. Polars out of the box works with the Arrow format and hence achieves much higher speeds.   Polars underlying code is implemented in Rust  and since it is a compiled language  unlike Python  which is interpreted  it has a speed advantage again. That is not the only reason  besides that there is memory safety and concurrency  which is better handled in Rust.   Production codeGreat API brings us back to the point of whether some should be using either library in production  which is another advantage for Polars. Pandas is not stable enough to be used in production  as it has been shown for years and discussed in the community. Many changes and underlying legacy code give so many pain points that it is not worth going with Pandas.   DependenciesI want to point out some of the advantages of Pandas as well  and those are dependencies  which are  in this case  a sword with two edges. Although this provides us with a lot of integration with libraries like Seaborn and matplotlib to achieve even better results  we are stuck with Pandas and sometimes cant move away from the library.   As mentioned  Polars primarily depends on the Arrow data format  which provides a high-performance  in-memory columnar data structure. This reduced dependency chain contributes to Polars overall performance and flexibility  as it avoids potential compatibility issues and overhead associated with managing multiple external libraries.   CommunityThe dependency problem will be solved as the community grows over time in this direction of clean code and efficiency  but it takes time. That is another advantage for Pandas because it has existed for so long.   With an increasing number of developers and data scientists adopting Polars for their projects  the ecosystem is expanding at an accelerated pace. While Pandas has a significant head start  the momentum behind Polars suggests that it will quickly close the gap in community size  resources  and available tools  positioning itself as a strong competitor in the data manipulation landscape. Still  this time  we are going in the right direction.   Switching from Pandas to PolarsTransitioning from Pandas to Polars can be a smooth process for many users due to the similar DataFrame structure and familiar Python syntax. While there are differences in API and functionality  Polars performance benefits  especially for large datasets  often outweigh the initial learning curve. Many everyday Pandas operations have direct equivalents in Polars  and the growing community provides ample resources and support to aid in the migration. However  for complex workflows heavily reliant on Pandas-specific features  a gradual adoption approach or hybrid use of both libraries might be necessary.   ConclusionStarting your Data Science journey with Polars can be good  but you will discover that many Stackoverflow questions and discussion forums are still focused on Pandas. Getting the right mindset from the get-go is vital so that Polars can be very beneficial later on as the starting point.   Switching from Pandas to Polars is also great  so going with Polars right now would benefit the project and developers working on the code.   That is all for today! If you have any questions  please send them my way!


From Solo Notebooks to Collaborative Powerhouse: VS Code Extensions for Data Science and ML Teams

In this article  we will explore the essential VS Code extensions that enhance productivity and collaboration for data scientists and machine learning (ML) engineers. We will discuss why VS Code may be a superior choice compared to Jupyter Notebooks  especially in team settings.   The Essence of Collaboration: From an Individual Working Environment to a Collaborative Data Science Environment.Why VS Code might be better for many data scientists and ML engineers than Jupyter Notebook.Essential VS Code Extensions for Data Scientists and ML Engineers.Factors Influencing the Choice Between Jupyter Notebooks and VS CodeHow to find new extensions for vs code for data science and machine learning.Conclusion.My story (The Shift from Jupyter Notebooks to VS Code)Throughout early to mid-2019  when I started my data science career  Jupyter Notebooks were my constant companions. Because of its interactive features  its ideal for learning and teaching  prototypes  exploratory data analysis projects  and visualizations. Think of them as digital scratchpads perfect for participating in Kaggle and Zindi competitions  creating data visualizations  and working directly with the data.   But things got complicated when I landed my first real data science gig and transitioned into a team environment.   Imagine the sceneYou have spent hours crafting a beautiful analysis in your notebook  a perfect marriage of code  and insightful commentary. You share it with the team  brimming with excitement  only to be frustrated. They cannot replicate your stellar results because of environment inconsistencies  missing libraries  and many other reasons.   Sharing bulky zip files containing notebooks  scripts  and datasets became a logistical nightmare. Reproducing results on different machines felt like alchemy; it was a frustrating guessing game with a cryptic mix of environment variables and missing dependencies that could frustrate even the most solid or experienced data scientist.   Did I install that library in the right virtual environment again?   This wasnt uncommon. Many beginner data scientists  myself included back then  struggled with the shift from solo exploration to collaborative  production-ready workflows.   We are data wranglers at heart  not necessarily software engineers by training  and best practices for reproducibility can sometimes get pushed aside in the heat of exploration.   Well  it seems cool  but the above is a recipe for collaboration chaos.   This experience highlighted the importance of seamless collaboration and reproducibility in data science teams. As a result  I turned to VS Code  which offers a more robust environment for teamwork and adherence to software engineering principles.   In my case  I found a solution for a larger team setting: VS Code.   Having explored various IDEs  I confidently recommend VS Code as a better option for Jupyter Notebooks regarding collaboration  following software engineering principles as a data scientist and machine learning engineer  and working with teams.   Compelling reasons why VS Code might be a better choice for many data scientists and ML Engineers than Jupyter Notebook working in teamsHeres a comparison between VS Code and Jupyter Notebook for data scientists and ML engineers in a collaborative environment:   These differences highlight how VS Code  with its extensive customization and integration options  can be a more efficient choice for many data scientists and ML engineers compared to Jupyter Notebook.   In this section  we will learn about the VS code extensions that are essential to my workspace and adhere to key software engineering principles.   Heres a glimpse at the list:   PythonPylanceJupyterJupyter Notebook RendererGitlensPython IndentDVCError lensGitHub Co-pilotData WranglerZenML StudioKedroSandDance1. Python ExtensionThe Python extension is crucial for efficient development  providing functionalities such as:   Linting and Syntax Checking: Helps identify errors in your code.Debugging and Code Navigation: Streamlines the debugging process and allows easy navigation through your codebase.Auto-Completion and Refactoring: Enhances coding efficiency and readability.Unit Testing Integration: Facilitates testing practices within your projects.This extension also automatically installs Pylance  which enhances the experience when working with Python files and Jupyter Notebooks.   2. Jupyter ExtensionThe Jupyter extension integrates the power of Jupyter notebooks into VS Code  offering:   Faster Loading Times: Improves the responsiveness of notebooks.Seamless Integration: Allows you to work within the familiar VS Code environment while leveraging Jupyters capabilities.Support for Multiple Languages: Basic notebook support for various programming languages enhances versatility.3. Jupyter Notebook RendererThis Jupyter Notebook Renderer allows you to view the outputs of your code directly within VS Code  eliminating the need to switch between windows. It enables dynamic updates of charts and graphs  detailed image previews  and interactive data visualizations  significantly enhancing the data exploration experience.   4. Python IndentProper indentation is vital in Python programming. The Python Indent extension automates indentation management  ensuring that your code adheres to best practices. It highlights potential indentation errors as you code  promoting readability and maintainability.   5. DVC (Data Version Control)The DVC extension transforms VS Code into a centralized hub for all your machine learning experimentation needs. For data scientists and ML engineers  the road to breakthrough models is often paved with countless experiments and data iterations. Without proper management  this process can quickly spiral into chaos.   Key Features:Comprehensive Versioning: Beyond just data  DVC versions metadata  plots  models  and entire ML pipelines.Advanced Experiment Tracking: Record code  data  parameters  and metrics. Easily compare and identify top-performing models.User-Friendly Interface: Includes a dashboard  live tracking  and GUI-based data management.Large File Handling: Simplifies and streamlines versioning of large files  a common pain point in ML projects.Real-time Monitoring: Watch metrics evolve live  enabling rapid adjustments during training.6. Error LensError lens enhances the visibility of errors and warnings in your code  providing inline diagnostic messages. This feature helps developers catch issues early  making the development process more efficient and reducing the time spent debugging.   7. GitLensVersion control is essential for collaborative projects. Gitlens integrates Git functionality within VS Code  allowing you to visualize Git history  understand code authorship  and navigate through branches and commits. This extension simplifies collaboration and helps prevent potential conflicts.   8. Data WranglerThe Data Wrangler extension offers an interactive interface for exploring  cleaning  and visualizing data. It generates Python code using Pandas as you work  making data manipulation efficient and code-friendly. This tool is invaluable for preparing data for further analysis.   9. ZenML StudioZenML Studio is a new extension that simplifies working with ZenML for MLOps projects. It integrates seamlessly with VS Code  providing a smooth experience for managing machine learning workflows.   10. Live ShareLive Share enables real-time collaborative development  allowing team members to co-edit and debug code together. This feature enhances the traditional pair programming experience by allowing developers to maintain their preferred settings while collaborating.   11. KedroThe Kedro extension for Visual Studio Code integrates the powerful Kedro framework  enhancing project management and collaboration for data scientists and machine learning engineers.   Key FeaturesStreamlines the organization of code  data  and configurations within Kedro projects.Enhances teamwork by providing features that allow multiple users to work on the same project efficiently.Pipeline Visualization.Code Quality and Testing.12. SandDance:Perfect for both data novices and seasoned analysts  SandDance shines when youre facing a new dataset and need to quickly grasp its essence. Its ability to reveal relationships between variables and highlight trends makes it an invaluable tool for initial data exploration and hypothesis generation.   Factors Influencing the Choice Between Jupyter Notebooks and VS CodeWhile VS Code offers numerous advantages for data science teams  the optimal choice between Jupyter Notebooks and VS Code depends on various factors:   Team SizeSmall teams: Jupyter Notebooks can be sufficient for very small  closely-knit teams where communication is frequent and informal. The interactive nature can facilitate rapid prototyping and experimentation.   Large teams: VS Codes version control integration  code organization  and debugging capabilities become increasingly valuable as team size grows. It promotes code standardization and reduces the risk of errors.   Project ComplexitySimple projects: Jupyter Notebooks can handle exploratory data analysis and small-scale modeling projects effectively.   Complex projects: VS Codes structured approach  debugging tools  and integration with other development tools are better suited for large-scale  production-oriented projects with multiple dependencies and complex workflows.   Individual PreferencesInteractive exploration: Data scientists who prefer an interactive  exploratory style may lean towards Jupyter Notebooks.   Code-centric workflow: Those who prioritize code organization  reusability  and collaboration may find VS Code more appealing.   Ultimately  the best approach often involves a hybrid strategy  leveraging the strengths of both environments. VS Code stands out as an ideal environment for complex data science projects that involve development  testing  and deployment  providing robust tools for collaboration and version control while still allowing for the interactive exploration capabilities of Jupyter Notebooks.   Finding New ExtensionsTo stay updated on the latest VS Code extensions  follow these steps:   Visit the VS Code MarketplaceUse the filter options to explore categories like Data Science and Machine Learning.Sort by Date to find the newest extensions.ConclusionIn summary  adopting Visual Studio Code (VS Code) along with its diverse extensions can significantly enhance collaboration for data science and machine learning teams.   Transitioning from Jupyter Notebooks to VS Code is not just a change in tools; it signifies a shift towards software engineering best practices that improve teamwork  reproducibility  and project management.VS Codes features  including integrated version control and real-time collaboration tools  streamline workflows and minimize common collaborative challenges.   While Jupyter Notebooks excel in interactive exploration  VS Code offers a more structured approach suitable for complex projects. Ultimately  the decision between the two should align with the teams specific needs  but for those aiming for a more collaborative and organized workflow  VS Code proves to be a superior choice.   Connect with me on LinkedIn   Connect with me on Twitter


TensorFlow vs. PyTorch: Whats Better for a Deep Learning Project?

Deep learning. A subset of machine learning utilizing multilayered neural networks  otherwise known as deep neural networks. Allowing society to simulate the decision-making prowess the human brain possesses  deep learning exists within some of the AI applications we use in our lives today.   If youre getting started with deep learning  youll find yourself overwhelmed with the amount of frameworks. However  youll see two frameworks stand at the top: PyTorch and TensorFlow. Possessing their own strengths and weaknesses  both these frameworks are powerful deep learning tools. PyTorch powers Teslas autopilot feature and OpenAIs ChatGPT  while TensorFlow is used in Google search and Uber.   Both TensorFlow and PyTorch are both relied on heavily in research and commercial code. APIs and cloud computing platforms extend the usage of both frameworks. If both of them have so much support and usage  how do you decide which one to use? Lets answer that question.   TensorFlow is an end-to-end platform for machine learning  a prominent open-source library dedicated to accomplishing a wide range of machine and deep learning tasks. Developed by Google in 2015  TensorFlow boasts extensive capabilities  resulting in the tool being used often for research purposes or companies using it for their programming purposes. It can also be used in a variety of languages  such as Python  C++  JavaScript  and Java.   FunctionalityOne thing to note is the name TensorFlow tells you how youre going to work with this framework. The basic data structure for TensorFlow are tensors. A tensor is an algebraic object detailing the multilinear relationship between sets of algebraic objects with respect to a vector space. There are many types of tensors  with some of the most popular ones being scalars and vectors  the 2 simplest tensors.   Now  a big focus for TensorFlow is on production and scalability. It becomes obvious when you take a look at its robust architecture and enormous support for deploying models on a variety of platforms. Lets take a look at what other reasons makes TensorFlow so reliable for production and scalability.   Production:   1. TensorFlow Extended (TFX):   End-to-End Pipeline: Providing a variety of tools and libraries for production-ready machine learning pipelines  TFX takes care of the entire lifecycle from data ingestion and validation to model training  evaluation  and deployment.Component Integration: TFX has components such as TensorFlow Data Validation  Transform  Model Analysis  and Serving. All of these components work well together and ensure a reliable production workflow.2. TensorFlow Serving:   Model Deployment: TensorFlow serving was specifically reated for deploying machine learning models in production. Supporting features such as model versioning  it allows for updates to be implemented easily.High Performance: TensorFlow has been optimized for low-latency and high-throughput serving  making it suitable for real-time interference applications.3. TensorFlow Lite:   Edge Deployment: TensorFlow Lite allows for you to deploy your models on mobile and other embedded devices. Optimizing models for performance and resource usage  it ensures efficient performance on resource-constrained devices.Hardware Acceleration: In addition  it supports various hardware accelerators  such as GPUs and TPUs  allowing for a performance boost on edge devices.Scalability:   Distributed Training:Multi-GPU and Multi-TPU Support: TensorFlow allows for groups to train models across multiple GPUs and TPUs  decreasing training time.Multi-Machine Training: It also facilitates training across several machines  enabling the handling of very large datasets and complex models.2. Docker and Kubernetes:   Containerization: TensorFlow allows for its models to be containerized using Docker  making it significantly easier to deploy  scale  and manage applications in various environments.Orchestration: You can also use Kubernetes to create TensorFlow workloads  which enables the automatic scaling  management of containerized applications  and deployment.3. Cloud Integration:   Google Cloud AI Platform: Integrating well with the Google Cloud API  TensorFlow can provide managed services for training and serving models.Other Cloud Providers: TensorFlow works well with other cloud platforms such as AWS and Azure  supporting scalable deployment and training in cloud environments.From this  it becomes obvious how TensorFlow prioritizes production and scalability. Even with all of this functionality and support  TensorFlow has something else that makes users fall in love with it: Keras.   Keras is an open-source deep-learning framework with a popularity stemming from its user-friendly interface. A high-level  user-friendly API  Keras allows you to build  train  and deploy deep-learning models very minimal code.   In TensorFlow 2.0  Keras was added in to the TensorFlow package as tf.keras  making it officially an API of TensorFlow. This integration allows users to access the simplicity of Keras whilst also leverging the pwoer and flexibility that TensorFlow offers. Any of the advanced features of TensorFlow  such as custom training loops and the TensorFlow Data API can be utilized whilst using tf.keras.   Its also very easy for beginners to start with deep learning through tf.keras because of the simplicity. At the same time  it gives advanced users the flexibility to build more complicated models.   Keras brings more life to TensorFlow  giving it a significant boost in popularity when the API was introduced to it. Now  with all these features  it may look like TensorFlow is the clear choice. TensorFlow has so much support and flexibility for designing deep learning models  so why ishere a need to look at a different framework? Well the answer is quite simple. PyTorch offers a dynamic experience whilst designing your deep learning models. So  lets take a look at PyTorch.   What is PyTorchPyTorch is an open-source deep learning framework developed by Facebook and released in 2016. Facebook released the framework with the intention of matching the production of TensorFlow while making it easier to write code for models. Since python programmers found it easy to use  PyTorch gained popularity at a rapid rate. PyTorch has an emphasis on providing a high-level user friendly interface while possessing immense power and flexibility for any deep learning task.   FunctionalityLike TensorFlow  the unit of data for PyTorch remains the tensor. However  PyTorch is based on Torch  a framework designed for fast computations which was written in the language Lua. Torch provided implementations of deep learning algorithms and tools  which heavily inspired PyTorchs design and fucntionality.   Now although PyTorch has an emphasis on easy usage and readability  it retains the power needed for users to accomplish complicated deep learning tasks. This allows for beginnners to easily learn thew framework while allowing more advanced users to build more complex models. Lets take a look at a couple of ways PyTorch accomplishes this.   Comprehensive Libraries and Tools:Torchvision: Library that provides datasets  model architectures  and image transformations.TorchText: Library for natural language processing (NLP). Offers datasets  tokenizers  and pre-trained word vectors.TorchAudio: Library for audio processingPyTorch Lightning: Framework for structuring PyTorch code  which makes it easier to manage training loops and logging.2. Dynamic Computation Graphs:   Eager Execution: PyTorch builds computation graphs as operations are executed. This dynamic nature makes PyTorch more flexible  allowing for debugging and modification.Immediate Feedback: Since operations are executed immediately  PyTorch gives immediate feedback  which makes it easier to experiment with different architectures and strategies.3. Production-Ready:   TorchScript: Allows you to run PyTorch models independent of Python. Easier to deploy models in production environments.ONNX (Open Neural Network Exchange): PyTorch supports exporting models to the ONNX format  which allows for interoperability with other frameworks and deployment on other platforms.4. Research and Prototyping:   Flexibility: The dynamic nature of PyTorch makes it perfect for research and prototyping. Researchers can implement and test new ideas without being concerned about static-graph constraints.Active Community: PyTorch has an active community of researchers and developers who are constantly contributing to its development.5. Visualization and Debugging:   TensorBoard Integration: Integrating with TensorBoard allows PyTorch to access visualizations of training metrics  model graphs  and other information.Advanced Debugging Tools: The dynamic nature of PyTorch simplifies debugging  allowing people to use the standard Python debugging tools.Use CasesWeve talked about the individual strengths of PyTorch and TensorFlow  but what about their use cases? When it is it most appropriate to implement one or the other? The use cases for TensorFlow are:   Production Deployment: With components such as TensorFlow Serving and Lite  TensorFlow is very well-suited for deploying machine learning models in production. TensorFlow provides a high performance serving system for models while allowing the user to deploy on mobile and embedded devices.Large -Scale Machine Learning: TensorFlow has built-in support for training across several GPUs and machines. This makes it very suitable for large-scale machine learning tasks.Applications: TensorFlow integrates well with commercial and enterprise applications such as Google Cloud  where TensorFlow can use the AI Platform  BigQuery  and Cloud Storage.As for PyTorch  they are as follows:   Research: PyTorchs dynamic computation graph allows it work well for researching and prototyping purposes. This allows for more intuitive and flexible model development.Computer Vision and NLP: Utilizing torchvision with PyTorch  you will have access to tools for computer vision  including pre-trained models  datasets  and image transformations. TorchText offers datasets  tokenizers  and pre-trained embeddings for natural language processing.Education: As PyTorch follows Pythons syntax  it makes it very easy for beginners to learn and use. PyTorch is used in academic courses often.Concluding ThoughtsLets recap  TensorFlow and PyTorch are powerful frameworks for deep learning. TensorFlow is often used for deployment purposes  while PyTorch is used for research. Based on what your task is  you can then choose either PyTorch or TensorFlow.   However  dont just stop with learning just one of the frameworks. Try and learn both. Both have their weaknesses and strengths and for a task where PyTorch may not work  TensorFlow could. For a task where TensorFlow may struggle  PyTorch may excel. Both frameworks are great at what they do and have made machine learning and deep learning much more accessible for everyone. I hope you enjoyed this article  and thank you for reading it!


Building a Productized AI Chatbot for Credit Card Business

IntroductionImagine youre a customer with an urgent question about your credit card. You call customer support  but the wait time is long  and the process is frustrating. This is where our AI chatbot comes in  transforming how customer support is handled in the credit card business.   When building a smarter  faster  and more secure customer support system  I realized the need for a chatbot that isnt just a novelty but a practical  production-ready tool. This chatbot uses AI technologies to ensure its efficient  secure  and easy to deploy.   I used Azure OpenAI to create the chatbot because its the best tool for understanding and responding to customer questions. However  keeping customer information safe is also important. To do this  I added Amazon Comprehend Moderation to protect personal data.   To make the chatbot more practical  I added a PostgreSQL database for specific data queries in the credit card business.   Deploying this chatbot is straightforward thanks to Docker containers  making it easy to scale and manage. Using tools like ChainLit and ConversationBufferWindowMemory  the chatbot can maintain a conversational history. This provides an excellent and personalized customer experience.   In this post  I want to show how these technologies combine to create a powerful AI chatbot that transforms customer support for credit cards. This setup can also be easily adapted for other businesses  like retail.   Items and FrameworkLets dive into the AI chatbot for credit card customer support by breaking down the entire framework  piece by piece.   AzureOpenAI for Intelligent Responses:   I chose Azure OpenAI as the brain of our chatbot because it fits perfectly with our needs as an enterprise application. It works better with other Azure services like AZURE_AI_SEARCH_SERVICE and data storage  making integration smooth and efficient. In addition  Azure OpenAI has robust security features and meets industry standards  necessary for keeping our data safe and secure.   Embedding Models and Data Chroma for Information Retrieval:   We need to store and find all the information the chatbot might retrieve. Embedding models and data Chroma acts like a well-organized library where each piece of information has a unique code. This makes it easy to find and ensures coherent conversations.   I tested Azure AI Search Retriever with Azure data storage and found it performs excellently. However  using Chroma with local data provides better protection for sensitive information  crucial for enhanced data privacy in business.   This database PostgreSQL contains detailed information about our credit card products. Whenever the chatbot answers a customers question  it also provides real-time promotions about credit cards with no annual fee from our database. Additionally  you can extend many functionalities by using database queries  such as recommending specific credit cards to customers based on the information they provide.   Amazon Comprehend Moderation for Data Privacy:   Privacy is the priority in business AI applications  especially with financial info. Amazon Comprehend Moderation scans and protects sensitive data like social security numbers and addresses. I tested it  and its great at keeping information safe. This ensures we comply with privacy laws  and make users feel secure sharing their information.   ChainLit and ConversationBufferWindowMemory for Conversational Management:   ChainLit and ConversationBufferWindowMemory act like the chatbots memory  helping it keep track of ongoing chats. This is very important for customer support since it often involves follow-up questions. These tools let the chatbot remember the context  making interactions more personal and coherent.   Docker for Deployment and Management:   Finally  we need a reliable way to deploy and manage our chatbot. Docker is chosen for its ability to deploy the chatbot in containers. It can isolate environments and can be easily scaled while maintaining security.   Imagine our chatbot can only initially handle 100 user requests per day. As our user base grows and we receive 1 000 requests  Docker may quickly scale container instances to meet this increased demand without altering the underlying code.   To put it all together  imagine a flowchart that starts with a customer query. This query gets processed by Azure OpenAI (the brain)  which then retrieves the needed information from our organized library (embedding models and data Chroma). Before responding  our security guard (Amazon Comprehend Moderation) checks for any sensitive data. The chatbot with its memory (ChainLit and ConversationBufferWindowMemory) delivers a coherent response. And overseeing everything is Docker  helping the system run smoothly and grow as needed.   Code Explanation for the AI ChatbotIll walk through the code that powers our AI chatbot. I use Chainlit for the user interface  LangChain for the conversational flow and memory  and Amazon Comprehend for ensuring data security.   Lets break down the code block by block to understand how each component works together.   1. Setting Up Environment VariablesFirst  we set up the necessary environment variables for Azure OpenAI and Amazon Comprehend. These keys and endpoints are essential for authenticating our API requests.   import os # Azure OpenAI credentials os.environ[AZURE_OPENAI_API_KEY] = your_azure_openai_api_key os.environ[AZURE_OPENAI_ENDPOINT] = https://your_openai_endpoint/ OPENAI_API_KEY = your_azure_openai_api_key OPENAI_DEPLOYMENT_NAME = gpt4 MODEL_NAME = gpt-4 OPENAI_API_VERSION = 2024-03-01-preview2. Initializing the Chatbot ModelThen initialize the Azure OpenAI model which will generate responses to user queries. This model uses the credentials set earlier.   from langchain_openai import AzureChatOpenAI # Set up the Azure Chat OpenAI model chat_model = AzureChatOpenAI( openai_api_version=OPENAI_API_VERSION  azure_deployment=OPENAI_DEPLOYMENT_NAME  temperature=0 )3. Embedding Model and RetrieverNext  I set up the embedding model and the retriever using LangChain and Chroma. This enables the chatbot to search through a vector database. In addition  I created a function to fetch promotional credit card products from the table credit_card_products in PostgreSQL.   from langchain_openai import AzureOpenAIEmbeddings from langchain_chroma import Chroma from sqlalchemy import create_engine # Postgresql Database connection conn_str = 'postgresql://#####' engine = create_engine(conn_str) # Obtain promotional credit card products from Postgresql def fetch_promotional_cards(): try: query = SELECT * FROM credit_card_products WHERE annual_fee < 1 df_promotional_cards = pd.read_sql(query  engine) return df_promotional_cards except Exception as e: print(fError fetching promotional cards: {e}) return pd.DataFrame() # Initialize the embedding model emb_model = AzureOpenAIEmbeddings( deployment='textembedding3large'  model='text-embedding-3-large'  openai_api_key=OPENAI_API_KEY  azure_endpoint=https://#####.com/  openai_api_type=azure ) # Define the function to load the retriever def get_retriever(): loaded_vectordb = Chroma(persist_directory=path_to_chroma_db  embedding_function=emb_model) retriever = loaded_vectordb.as_retriever() return retriever chat_retriever = get_retriever()4. Managing Conversation ContextWe use ConversationBufferWindowMemory from LangChain to maintain conversational context. It allows the chatbot to keep track of previous interactions.   from langchain.memory import ConversationBufferWindowMemory # Set up the conversation memory chat_memory = ConversationBufferWindowMemory( k=5  memory_key=chat_history  input_key=question  output_key='answer'  return_messages=True )5. Amazon Comprehend for Data SecurityAmazon Comprehend Moderation is configured to scan and protect sensitive data. This ensures that any personally identifiable information (PII) in user queries is handled securely.   import boto3 from langchain_experimental.comprehend_moderation import ( AmazonComprehendModerationChain  BaseModerationConfig  ModerationPiiConfig  ModerationPromptSafetyConfig  ModerationToxicityConfig ) # Set up the Amazon Comprehend Moderation os.environ[AWS_ACCESS_KEY_ID] = #### os.environ[AWS_SECRET_ACCESS_KEY] = ###### # Initialize the Amazon Comprehend client comprehend_client = boto3.client(comprehend  region_name=us-east-1) # Define moderation configurations pii_labels = [SSN  DRIVER_ID  ADDRESS  'EMAIL'  'PHONE'  'CA_SOCIAL_INSURANCE_NUMBER'] pii_config = ModerationPiiConfig(labels=pii_labels  redact=True  mask_character=X) toxicity_config = ModerationToxicityConfig(threshold=0.5) prompt_safety_config = ModerationPromptSafetyConfig(threshold=0.5) moderation_config = BaseModerationConfig( filters=[pii_config  toxicity_config  prompt_safety_config] ) comp_moderation_with_config = AmazonComprehendModerationChain( moderation_config=moderation_config  client=comprehend_client  verbose=True )6. Defining System and Human Message TemplatesThen define templates for system and human messages to guide the chatbots interactions. This enables it to follow a structured approach.   from langchain.prompts import ChatPromptTemplate  HumanMessagePromptTemplate  SystemMessagePromptTemplate # Define system and human message templates system_template =  You are a virtual assistant for the Help Desk. Only answer questions related to credit card business. Include URL in your reply  and return the full URL for your source document. Do not include any email. Use the answers from the retrieved document first. If you cannot find the answer from the pieces of context  just say that sorry you don't know nicely. Do not try to make up an answer. All the personally identifiable information will be redacted with X. Ignore the personally identifiable information and answer generally. --------------- {context}  human_template = Previous conversation: {chat_history} New human question: {question}  messages = [ SystemMessagePromptTemplate.from_template(system_template)  HumanMessagePromptTemplate.from_template(human_template) ] qa_prompt = ChatPromptTemplate.from_messages(messages)7. Setting Up the Conversational Retrieval ChainThis step sets up the conversational retrieval chain that ties everything together  using the model  retriever  memory  and moderation components.   from langchain.chains import ConversationalRetrievalChain # Initialize the conversational retrieval chain qa = ConversationalRetrievalChain.from_llm( llm=chat_model  chain_type='stuff'  retriever=chat_retriever  memory=chat_memory  return_source_documents=True  combine_docs_chain_kwargs={prompt: qa_prompt} )8. Chainlit Integration for Chatbot UIFinally  integrate Chainlit to handle user interactions. This provides a user-friendly interface.   import chainlit as cl @cl.on_chat_start async def on_chat_start(): msg = cl.Message(content=Hello  this is AI powered helpdesk  feel free to ask me any questions!) await msg.send() cl.user_session.set(chain  qa) @cl.on_message async def main(message: cl.Message): chain = cl.user_session.get(chain) cb = cl.AsyncLangchainCallbackHandler( stream_final_answer=True ) # Force final answer if necessary cb.answer_reached = True res = await chain.acall(message.content  callbacks=[cb]) answer = res[answer] source_documents = res[source_documents] # type: List[Document] text_elements = [] # type: List[cl.Text] if source_documents: for source_idx  source_doc in enumerate(source_documents): source_name = fsource_{source_idx} text_elements.append( cl.Text(content=source_doc.page_content  name=source_name  display=side) ) source_names = [text_el.name for text_el in text_elements] answer += f\n\nSources: {'  '.join(source_names)} else: answer += \n\nNo sources found # Fetch promotional credit card products df_promotional_cards = fetch_promotional_cards() promotional_cards_info = df_promotional_cards.to_dict(orient='records') # Append promotional cards information to the answer if promotional_cards_info: answer += \n\nPromotional Credit Cards with No Annual Fee:\n for card in promotional_cards_info: answer += f- {card['card_name']} (Credit Limit: {card['credit_limit']}  Cashback: {card['cashback']}%  Sign-Up Bonus: {card['sign_up_bonus']} points)\n await cl.Message(content=answer  elements=text_elements).send()By combining Chainlit for the UI  LangChain for conversation management  a PostgreSQL database for detailed credit card information  and Amazon Comprehend for data security  we can create a professional and robust chatbot solution.   Deploying the AI Chatbot Using DockerTo make our AI chatbot professional and production-ready  we need to deploy it using Docker. Docker allows us to package our application with all its dependencies into a container that can run consistently on any system. Heres a detailed guide on how to build and deploy our chatbot using Docker.   Dockerfile and RequirementsFirst  lets look at the Dockerfile and the requirements.txt file  which are essential for building our Docker image.   Dockerfile:   # Stage 1 - Install build dependencies FROM python:3.11-slim AS builder WORKDIR /app ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 RUN apt-get update && apt-get install -y \ build-essential \ curl \ software-properties-common \ git \ libpq-dev \ && rm -rf /var/lib/apt/lists/* RUN python -m venv /opt/venv ENV PATH=/opt/venv/bin:$PATH COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Stage 2 - Copy only necessary files to the runner stage FROM python:3.11-slim ARG FILENAME ARG PORT=8000 ENV FILENAME=${FILENAME} ENV PORT=${PORT} WORKDIR /app COPY --from=builder /opt/venv /opt/venv ENV PATH=/opt/venv/bin:$PATH COPY $FILENAME . COPY chainlit.json . COPY static ./static EXPOSE ${PORT} CMD [sh  -c  python -m chainlit run ${FILENAME} --port=${PORT} -w]requirements.txt:   chainlit langchain boto3 langchain_experimental langchain_openai langchain_community langchain_chroma flask psycopg2 SQLAlchemyBasics of Dockerfile and requirements.txtStage 1  Build Dependencies:   Base Image: We use the python:3.11-slim image as the base.Working Directory: Sets the working directory to /app .Environment Variables: Disables bytecode generation and ensures unbuffered output.System Dependencies: Installs build-essential tools  curl  and git.Virtual Environment: Creates a virtual environment and updates the PATH.Dependencies Installation: Copies requirements.txt and installs the Python dependencies.Stage 2  Runner Stage:   Base Image: Uses the same python:3.11-slim base image.Arguments and Environment Variables: Sets up arguments and environment variables for the filename and port.Working Directory: Sets the working directory to /app .Copy Files: Copies the virtual environment  application files  and necessary directories.Expose Port: Exposes the specified port.Command: Runs the Chainlit application using the specified filename and port.requirements.txt:   Lists all the necessary Python packages for the chatbot  ensuring they are installed in the Docker image.Steps to Build and Deploy the Docker ContainerCreate Dockerfile and requirements.txt:   Ensure both files are in your project root directory.Build the Docker Image:   Use the Docker build command to create the Docker image.docker build -t ai-chatbot --build-arg FILENAME=your_chatbot_script.py .Run the Docker Container:   Use the Docker run command to start a container from the image.docker run -d -p 8000:8000 --name ai-chatbot-container ai-chatbotAccess the Chatbot:   Once the container is running  you can access the chatbot by navigating to http://localhost:8000 in your web browser.Deploying the AI chatbot using Docker is necessary for ensuring it is production-ready  scalable  and consistent across different environments.   Testing Azure Cognitive Search for Enhanced PerformanceIn this project  I also tested Azure Cognitive Search as an alternative to the embedding-based Chroma search. Azure Cognitive Search is a cloud search service that provides indexing and querying capabilities.   Azure Cognitive Search Code:   from langchain_community.vectorstores.azuresearch import AzureSearch # Set up the Azure AI Search Retriever if using AZURE_COGNITIVE_SEARCH_SERVICE os.environ[AZURE_COGNITIVE_SEARCH_SERVICE_NAME] = your_search_service_name os.environ[AZURE_AI_SEARCH_SERVICE_NAME] = your_ai_search_service_name os.environ[AZURE_COGNITIVE_SEARCH_INDEX_NAME] = your_index_name os.environ[AZURE_AI_SEARCH_INDEX_NAME] = your_index_name os.environ[AZURE_COGNITIVE_SEARCH_API_KEY] = your_search_api_key os.environ[AZURE_AI_SEARCH_API_KEY] = your_ai_search_api_key # Initialize the Azure Cognitive Search retriever search_retriever = AzureSearch( service_name=os.getenv(AZURE_COGNITIVE_SEARCH_SERVICE_NAME)  index_name=os.getenv(AZURE_COGNITIVE_SEARCH_INDEX_NAME)  api_key=os.getenv(AZURE_COGNITIVE_SEARCH_API_KEY) )Advantages:   Performance: Azure Cognitive Search shows better performance in speed and accuracy when retrieving relevant documents compared to the embedding-based Chroma search.Scalability: Azure Cognitive Search can handle large-scale search queries efficiently. This makes it better for enterprise applications.Integration: Azure Cognitive Search integrates better with other Azure services. This provides a cohesive environment for querying data.Drawbacks:   Cost: Azure Cognitive Search can be more expensive compared to using local embedding-based solutions like Chroma.Complexity: Setting up and configuring Azure Cognitive Search can be more complex  requiring additional knowledge and management.Data Storage:   Azure: Storing RAG (Retrieval-Augmented Generation) data in Azure provides benefits such as high availability  redundancy  and security. However  it may also incur higher costs and dependency on cloud infrastructure.Chroma: Using Chroma with local storage can be more cost-effective and allows greater control over data. However  it may not scale as efficiently as Azure Cognitive Search.As usual  you can find the relevant code in the following GitHub repository: https://github.com/datalev001/chatbot_chainlit   Final ThoughtsUsing Docker to deploy our AI chatbot with tools like Chainlit  LangChain  and Amazon Comprehend makes it professional  scalable  and secure. This setup handles complex interactions  keeps track of conversations  and protects sensitive info.   Adding a PostgreSQL database lets the chatbot give personalized responses and real-time promotions  making it a great business tool. It uses specific data to provide tailored support and recommendations.   Testing Azure Cognitive Search showed its great for big queries  though its more complex and costly than local solutions like Chroma.   By setting up the environment  integrating advanced models  and deploying with Docker  we can create a solid AI helpdesk system.