Upload 35 files

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+database/document_store.pkl filter=lfs diff=lfs merge=lfs -text
+data/inc_df_v6_small_4.csv filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,21 @@
+# DS Store
+data/.DS_Store
+.DS_Store
+
+__pycache__/
+
+# sandbox
+sandbox.py
+
+# Enviroment Files
+.env
+.incenv
+
+archive/
+
+.env_example
+.pre-commit-config.yaml
+incenv/
+
+
+

.streamlit/config.toml

@@ -0,0 +1,7 @@
+[theme]
+primaryColor="#b50d1c"
+backgroundColor="#FFFFFF"
+secondaryBackgroundColor="#eceded"
+textColor="#262730"
+font="sans serif"
+base="light"

README.md

@@ -1,12 +1,82 @@
----
-title: NegotiateAI
-emoji: 🏆
-colorFrom: green
-colorTo: green
-sdk: streamlit
-sdk_version: 1.38.0
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# INC Plastic Treaty App
+
+## Set up Development
+
+### Install poetry environment
+
+Please install [poetry]("https://python-poetry.org/docs/") first before executing the following commands
+
+```bash
+poetry install
+``````
+
+#### Install pre-commit hooks
+
+```bash
+pre-commit install
+pre-commit
+``````
+
+
+#### Start app
+
+- excute the following command with the respective version
+
+```bash
+streamlit run src/app/v1/app.py
+``````
+
+## Data Overview
+
+- taxonomy_related data
+  - ```data/authors_taxonomy.json```: raw countries taxonomy
+  - ```data/draft_cat_taxonomy.json```: draft cat taxonomy.json
+  - ```data/authors_filter.json```: processed taxonomy for frontend filtering
+  - ```data/draftcat_taxonomy_filter.json```: processed taxonomy for frontend filtering
+  - ```data/inc_df_v6_small.csv```: processed data of scraping
+  - ```data/inc_df.csv```: data for document Storage
+  - ```data/taxonomies.txt```: raw taxonomies collection
+- application related data
+  - ```database/document_store.pkl```: Document Store
+  - ```database/document_store.pkl```: Meta data of the document store with countries and draft labs as columns
+
+## Code Structure
+
+### Data Preprocessing
+
+- ```src/data_processing/document_store_data.py```: Generates the data for the document store with last processing steps
+- ```src/data_processing/get_meta_data_filter.py```: Generates the meta data from document store for the filtering in the frontend
+- ```src/data_processing/taxonomy_processing.py```: Transforms the taxonomies for the filters in the frontend.
+
+### Frontend
+
+- ```src/app/```: versions of app
+- ```src/utils```: The utils folder contains functions used in the frontend application. Please check the imports to see which functions are used.
+- ```styles```: css styles for apps
+- ```.streamlit```: Basic Theme of Frontend
+- Some Settings and Changes related to the Spaces App have to be done directly in the respective Streamlit App.
+
+### Backend
+
+- ```src/document_store/document_store.py```: Generates the document store
+- ```src/rag/pipeline.py```: RAG Pipeline
+- ```src/rag/prompt/prompt_template.yaml```: Prompt Template for RAG
+
+## Changes Document Storage 
+
+- **Step 1**: Delete the ```database/document_store.pkl``` and the ```database/meta_data.csv``` files or give them a new name if you want to keep them until a successful change of the document storage.
+- **Step 2**: Run the script ```data_processing/document_store_data.py```. This changes update the ```data/inc_df.csv``` data. If you need to make changes in the ```data/inc_df_small_v6.csv```, they have to made either manually or you have to implement a new script with the changes.
+- **Step 3***: Run the script ```data_processing/get_meta_data_filter.py```. This will save a new document store and the meta_data from the document store.
+- **Step 4**: If the changes affect also the taxonomy then you need to update the taxonomies as well. To do so, first update manually ```data/authors_taxonomy.json``` and ```data/draftcat_taxonomy.json```. Then run the script ```src/data_processing/taxonomy_processing.py```.
+- Frequent Bugs after changes of the data:
+  - new country or draft lab category or changes in the naming. Solution: Check the taxonomy files and update the ```src/data_processing/taxonomy_processing.py```
+  - countries with special characters. Solution: Check ```data/inc_df_small_v6.csv``` if the ```data_processing/document_store_data.py``` fails. Check the ```database/meta_data.py```if the frontend application fails after changes and the taxonomies + filters.
+  
+## Changes App
+
+- Please always check changes in the app locally before pushing to the respective Spaces App.
+- To check the Spaces App locally you can clone it like Git Repositories. Do avoid making changes directly in the interface of the Spaces App.
+- Also if you copy changes from git to Spaces only copy the files where you have made changes. You need to make some adjustment before you copy the files:
+  - Please remove all OPENAI_KEYS from the app.py file and the the pipeline.py files.
+  - Make sure you added the following at the ```OPENAI_API_KEY = os.environ.get("OPEN_API_KEY")``` in the pipeline.py file
+  - Remove from app.py and pipeline.py ```src``` from imports. Otherwise you will get a ModuleNotFoundError.

data/authors_filter.json

@@ -0,0 +1,518 @@
+{
+    "Members - Countries": [
+        "Algeria",
+        "Angola",
+        "Antigua And Barbuda",
+        "Argentina",
+        "Armenia",
+        "Australia",
+        "Azerbaijan",
+        "Bahrain",
+        "Bangladesh",
+        "Barbados",
+        "Benin",
+        "Bosnia And Herzegovina",
+        "Brazil",
+        "Burkina Faso",
+        "Cambodia",
+        "Cameroon",
+        "Canada",
+        "Chile",
+        "China",
+        "Colombia",
+        "Comoros",
+        "Cook Islands",
+        "Costa Rica",
+        "Cuba",
+        "Cote D'Ivoire",
+        "Democratic Republic Of The Congo",
+        "Dominican Republic",
+        "Ecuador",
+        "Egypt",
+        "El Salvador",
+        "Equatorial Guinea",
+        "Eritrea",
+        "Eswatini",
+        "Ethiopia",
+        "Fiji",
+        "France",
+        "Gabon",
+        "Gambia",
+        "Georgia",
+        "Germany",
+        "Ghana",
+        "Grenada",
+        "Guatemala",
+        "Guinea",
+        "Guinea Bissau",
+        "Honduras",
+        "Iceland",
+        "India",
+        "Indonesia",
+        "Iraq",
+        "Islamic Republic Of Iran",
+        "Israel",
+        "Italy",
+        "Jamaica",
+        "Japan",
+        "Jordan",
+        "Kazakhstan",
+        "Kenya",
+        "Kiribati",
+        "Kuwait",
+        "Libya",
+        "Madagascar",
+        "Malawi",
+        "Malaysia",
+        "Maldives",
+        "Mali",
+        "Marshall Islands",
+        "Mauritius",
+        "Mexico",
+        "Micronesia",
+        "Moldova",
+        "Monaco",
+        "Mongolia",
+        "Montenegro",
+        "Morocco",
+        "Nauru",
+        "Nepal",
+        "New Zealand",
+        "Niger",
+        "Nigeria",
+        "North Macedonia",
+        "Norway",
+        "Oman",
+        "Pakistan",
+        "Palau",
+        "Panama",
+        "Papua New Guinea",
+        "Paraguay",
+        "Peru",
+        "Philippines",
+        "Qatar",
+        "Republic Of Congo",
+        "Republic Of Korea",
+        "Republic Of Moldova",
+        "Romania",
+        "Russian Federation",
+        "Rwanda",
+        "Samoa",
+        "Saudi Arabia",
+        "Senegal",
+        "Seychelles",
+        "Sierra Leone",
+        "Singapore",
+        "Solomon Islands",
+        "Somalia",
+        "South Africa",
+        "Sri Lanka",
+        "State Of Palestine",
+        "Switzerland",
+        "Syrian Arab Republic",
+        "Thailand",
+        "Timor-Leste",
+        "Togo",
+        "Tonga",
+        "Trinidad And Tobago",
+        "Tunisia",
+        "Turkey",
+        "Tuvalu",
+        "Uganda",
+        "Ukraine",
+        "United Arab Emirates",
+        "United Kingdom Of Great Britain And Northern Ireland",
+        "United Republic Of Tanzania",
+        "United States Of America",
+        "Uruguay",
+        "Venezuela",
+        "Vietnam",
+        "Yemen",
+        "Zambia",
+        "Zimbabwe"
+    ],
+    "Members - International and Regional State Associations": [
+        "Asia-Pacific States",
+        "European Union (EU) And Its 27 Member States",
+        "Federated States Of Micronesia",
+        "Gulf Cooperation Council",
+        "High Ambition Coalition",
+        "Latin American And Caribbean Group (GRULAC)",
+        "Like Minded Countries",
+        "Pacific Small Island Developing States (PSIDS)",
+        "Alliance Of Small Island States (AOSIS)",
+        "The Federated States Of Micronesia",
+        "The Group Of African States",
+        "Western European And Other States",
+        "Coordinating Body on the Seas of East Asia (COBSEA)"
+    ],
+    "Intergovernmental Negotiation Committee": [
+        "Executive Secretary Of The INC",
+        "Intergovernmental Negotiation Committee"
+    ],
+    "Observers and Other Participants": [
+        "10YFP Secretariat (One Planet Network)",
+        "ABIPLAST - Brazilian Association Of The Plastic Industry",
+        "ACTS Organization",
+        "ASTM International (ASTM)",
+        "AWTAD Anti-Corruption Organization",
+        "Aaina",
+        "Action Et Education Pour Tous (AEPT)",
+        "Action On Smoking And Health (ASH)",
+        "Africa Climate And Environment Foundation (ACEF)",
+        "African Alliance For Health Research Economic Development",
+        "African Environmental Network",
+        "African Petroleum Producers\u2019 Organization (APPO)",
+        "Aguaclara",
+        "Alianza Basura Cero",
+        "Alianza Latinoamericana De Asociaciones De La Industria De Alimentos Y Bebidas (ALAIAB)",
+        "All-China Environment Federation",
+        "Alliance Panafricaine Multi-Acteurs Sur La Pollution Plastique",
+        "Alliance Pour Le Controle Du Tabac En Afrique (ACTA)",
+        "American Chemistry Council",
+        "American Fuel and Petrochemical Manufacturers (AFPM)",
+        "And Colectivo Jaguares De Nuestra Madre Tierra",
+        "Aotearoa Plastic Pollution Alliance (APPA)",
+        "Arctic Monitoring And Assessment Programme (AMAP)",
+        "Armenian Women For Health And Healthy Environment",
+        "Arnika",
+        "Asia Indigenous Peoples Pact And Indigenous Peoples",
+        "Asian Marine Conservation Association",
+        "Asociaci\u00f3n Ecol\u00f3gica Santo Tom\u00e1s",
+        "Asociaci\u00f3n Sustentar",
+        "Association For Promoting Sustainability In Campuses And Communities (APSCC)",
+        "Association For Rural Area Social Modification, Improvement And Nestling (ARASMIN)",
+        "Association For Supporting The SDGs For The United Nations",
+        "Association Institute Of Total Environment (INTEV)",
+        "Association Nationale Du Civisme",
+        "Association Of Leadership You-Lean",
+        "Association Of Plastic Recyclers",
+        "Association Of Solidarity Through Humanitarian Imperative Actions International (ASHIA)",
+        "Association Pour L'Integration Et La Developpement Durable Au Burundi",
+        "Australian Packaging Covenant Organisation (APCO)",
+        "Azul",
+        "BAN Toxics",
+        "BVRio",
+        "Barranquilla + 20",
+        "Basel Action Network (BAN)",
+        "Beijing Greenovation Institute For Public Welfare Development (GHub)",
+        "Brazilian Chemical Industry Association",
+        "Break Free From Plastic",
+        "Breathe Free Detroit",
+        "Bridgers Association Cameroon",
+        "Bureau Of International Recycling",
+        "Business And Industry Major Group Presented By World Business Council For Sustainable Development",
+        "Business Coalition For A Global Plastics Treaty Convened By The Ellen MacArthur Foundation And WWF, In Collaboration With Aligned Businesses And Financial Institutions, And Supported By NGO Partners",
+        "CAIRPLAS, C\u00e1mara Argentina De La Industria De Reciclados Plasticos",
+        "CDP",
+        "CEMPRE Colombia \u2013 Compromiso Empresarial Para El Reciclaje",
+        "CESTA",
+        "Carrizo Comecrudo Tribe",
+        "Center For Biological Diversity",
+        "Center For Coalfield Justice",
+        "Center For International Environmental Law (CIEL)",
+        "Center For International Law And Governance",
+        "Center For Islamic Studies Of Universitas Nasional, Jakarta",
+        "Center For Oceanic Awareness, Research, And Education (COARE)",
+        "Center For Public Health And Environmental Development (CEPHED)",
+        "Centre De Recherche Et D\u2019Education Pour Le D\u00e9veloppement (CREPD)",
+        "Centre D\u2019Accompagnement Des Alternatives Locales De D\u00e9veloppement (Caald)",
+        "Centre For Environmental Justice",
+        "Centre For Human Rights And Climate Change Research",
+        "Centre For Science And Environment (CSE)",
+        "Centre International De Droit Compar\u00e9 De L\u2019environnement",
+        "Centre International De Droit Compar\u00e9 De L\u2019environnement",
+        "Chemical And Allied Industries\u2019 Association (CAIA)",
+        "Chia Funkuin Foundation",
+        "Children And Youth Major Group (CYMG)",
+        "Children\u2019s Environmental Health Foundation",
+        "China Biodiversity Conservation And Green Development Foundation (CBCGDF)",
+        "Circular Economy For Flexible Packaging In Europe Initiative (CEFLEX)",
+        "Citeo",
+        "Citizen Consumer And Civic Action Group (CAG)",
+        "Civil Society And Rights Holder Coalition",
+        "Civil Society Organizations In Africa",
+        "Civil Society Organizations In Asia Pacific",
+        "Civil Society Organizations In Latin America",
+        "Co-habiter",
+        "Comision Centro Americana",
+        "Comit\u00e9 National Contre Le Tabagisme",
+        "Community Action Against Plastic Waste (CAPws)",
+        "Congregations Of St Joseph",
+        "Consciente Colectivo",
+        "Consumer Goods Forum",
+        "Consumers International",
+        "Contact Group 1",
+        "Contact Group 2",
+        "Convention On Biological Diversity (CBD)",
+        "Corporate Accountability",
+        "Council For Scientific And Industrial Research",
+        "C\u00e1mara De La Industria Qu\u00edmica Y Petroqu\u00edmica (CIQyP\u00ae), Miembro Del Concejo Internacional De Asociaciones Qu\u00edmicas (ICCA) Y EURECA",
+        "Danimer Scientific",
+        "EDANA And INDA",
+        "EPS Branchen-en Del Af Plastindustrien (Denmark)",
+        "EPS Industry Alliance",
+        "EURECA",
+        "Earth Day",
+        "Earth Law Center",
+        "Ecoplas",
+        "Ecoproject Partnership",
+        "Ellen MacArthur Foundation",
+        "Endocrine Society",
+        "Engineers Australia",
+        "Entidad Especializada En Pl\u00e1sticos Y Medio Ambiente Para Una Econom\u00eda Circular (ECOPLAS)",
+        "Entidades Unidas Reafirmando La Econom\u00eda Circular En Argentina (EURECA)",
+        "Environmental And Social Development Organization (ESDO)",
+        "Environmental Coalition On Standards (ECOS)",
+        "Environmental Development Association (FASEEL)",
+        "Environmental Investigation Agency",
+        "European Bioplastics (EUBP)",
+        "European Manufacturers Of EPS",
+        "European Manufacturers Of Expanded Polystyrene (EUMEPS)",
+        "Expanded Polystyrene Australia",
+        "Fauna and Flora International",
+        "Fauna and Flora International And Zoological Society Of London (ZSL)",
+        "Fenceline Watch",
+        "First Modern Agro Tools Common Initiative Group ( FI.MO.AT.C.I.G)",
+        "Fondation De La Mer",
+        "Food And Agriculture Organization Of The United Nations (FAO)",
+        "Food And Livestock Initiative (FLI Asbl)",
+        "Forum On Trade, Environment and The SDGs (TESS)",
+        "Foundation Of Fokus Nexus Tiga (Nexus3 Foundation)",
+        "French Water Partnership (Partenariat Fran\u00e7ais Pour L\u2019eau)",
+        "Friends World Committee For Consultation (FWCC)",
+        "Fronteras Comunes",
+        "Fundacion Avina",
+        "Fundaci\u00f3n Ambiente Y Recursos Naturales",
+        "Fundaci\u00f3n Interamericana Del Coraz\u00f3n (FIC)",
+        "GRID-Arendal",
+        "Galapagos Conservation Trust",
+        "Gallifrey Foundation",
+        "Geneva Cities Hub (GCH)",
+        "Gerakan Indonesia Diet Kantong Plastik (GIDKP) - The Indonesia Plastic Bag Diet Movement",
+        "Global Alliance For Incinerator Alternatives (GAIA)",
+        "Global Alliance On Health And Pollution (GAHP)",
+        "Global Cement And Concrete Association",
+        "Global Council For Science And The Environment (GCSE)",
+        "Global Plastics Policy Centre, University Of Portsmouth",
+        "Global Youth Coalition On Plastic Pollution (GYCPP)",
+        "Green Africa Youth Organization",
+        "Greenpeace International",
+        "GroundWork South Africa (GroundWorkSA)",
+        "Haitelmex Foundation",
+        "Hasiru Dala In Collaboration With Eleven Other Civil Society Organizations",
+        "Health And Environment Justice Support (HEJSupport)",
+        "Health Care Without Harm (HCWH)",
+        "Healthy Hospitals Project - PHS",
+        "Human Rights Watch",
+        "ICLEI - Local Governments For Sustainability",
+        "India Institute For Critical Action Centre In Movement (CACIM)",
+        "India Water Foundation",
+        "India Youth For Society",
+        "Indigenous Caucus",
+        "Indigenous Peoples And Their Communities Major Group",
+        "Indigenous Peoples Representatives",
+        "Indonesian Centre For Environmental Law (ICEL)",
+        "Innovazing Vision",
+        "Institute For Sustainable Development And Research (ISDR)",
+        "Integrated Strategies Forum",
+        "Interamerican Heart Foundation",
+        "International Air Transport Association",
+        "International Alliance Of Waste Pickers (IAWP)",
+        "International Alliance Of Waste-pickers",
+        "International Atomic Energy Agency (IAEA)",
+        "International Center Of Comparative Environmental Law (CIDCE)",
+        "International Centre For Environmental Education And Community Development (ICENECDEV)",
+        "International Chamber Of Commerce (ICC)",
+        "International Council Of Beverages Associations (ICBA)",
+        "International Council Of Chemical Associations (ICCA)",
+        "International Knowledge Hub Against Plastic Pollution (IKHAPP)",
+        "International Labour Office",
+        "International Labour Organization (ILO)",
+        "International Maritime Organization (IMO)",
+        "International Medical Crisis Response Alliance (IMCRA)",
+        "International Movement For Advancement Of Education Culture Social and Economic Development (IMAESED)",
+        "International Network For Bamboo And Rattan (INBAR)",
+        "International Organization For Standardization (ISO)",
+        "International Pollutants Elimination Network (IPEN)",
+        "International Science Council (ISC)",
+        "International Society Of Doctors For The Environment (ISDE)",
+        "International Solid Waste Association (ISWA)",
+        "International Trade Union Confederation (ITUC)",
+        "International Union For Conservation Of Nature And Natural Resources (IUCN)",
+        "Inuit Circumpolar Council (ICC)",
+        "Japan Clean Ocean Material Alliance (CLOMA)",
+        "King Abdullah Petroleum Studies And Research Center (KAPSARC)",
+        "Krityanand UNESCO Club, Jamshedpur",
+        "La Grande Puissance De Dieu",
+        "Latin American Organizations - Alianza Basura Cero",
+        "Ligue Camerounaise Des Droits De L'Homme",
+        "Litter4tokens South Africa NPO",
+        "Local And Subnational Government Working Group",
+        "Local Authorities",
+        "Loop",
+        "Major Alliance Education Centre (MAEC)",
+        "Major Group For Children And Youth",
+        "MarViva",
+        "Marine Ecosystems Protected Areas (MEPA) Trust",
+        "Members Of Microplastics Working Group",
+        "Mexican Network Of Ecological Action",
+        "Minderoo Foundation",
+        "Ministry Of Environment And Wildlife - Southwest State Of Somalia",
+        "Moms Clean Air Force",
+        "Multifaith Action Group On Pollution",
+        "NGO Major Group",
+        "NORCE On Behalf Of The North Atlantic Microplastic Centre (NAMC)",
+        "National Old Folks Of Liberia (NOFOL)",
+        "National Retail Association (NRA)",
+        "Natural Resources Defense Council (NRDC)",
+        "Neste",
+        "Nexus For Health, Environment, And Development (Nexus3) Foundation",
+        "Nipe Fagio",
+        "No Balloon Release Australia",
+        "No More Butts",
+        "Norwegian Academy Of International Law (NAIL)",
+        "Norwegian Institute For Water Research",
+        "Norwegian Institute For Water Research (NIVA)",
+        "Norwegian Research Centre (NORCE)",
+        "ONG Jeunesse Active De Guin\u00e9e (JAG)",
+        "Occidental Arts And Ecology Center (OAEC)",
+        "Ocean Conservancy",
+        "Ocean Recovery Alliance",
+        "Ocean. Now",
+        "OceanCare",
+        "OceanCare Global Ghost Gear Initiative",
+        "Office Of The UN High Commissioner For Human Rights (OHCHR)",
+        "OpenOceans Global",
+        "Organisation For Economic Co-operation And Development (OECD)",
+        "Organization Of Arab Petroleum Exporting Countries (OAPEC)",
+        "Organization Of The Petroleum Exporting Countries (OPEC)",
+        "Our Sea Of East Asia Network (OSEAN)",
+        "Out For Sustainability",
+        "PCX Solutions (HOPEx Environment Group, Inc)",
+        "Pacific Environment And Resources Center (Pacific Environment)",
+        "Pan American Neuroendocrine Society",
+        "Partnerships For Change",
+        "Paryavaran Mitra",
+        "PetStar",
+        "Planeteer Alliance And Captain Planet Foundation",
+        "Plastalliance - Alliance Plasturgie Et Composites Du Futur",
+        "Plastic Change",
+        "Plastic Free Foundation",
+        "Plastic Free Future",
+        "Plastic Oceans Australasia",
+        "Plastic Pollution Coalition",
+        "Plastics Federation Of South Africa",
+        "Plastics Industry Association",
+        "PlasticsEurope",
+        "Plasticulture",
+        "ProDelphinus",
+        "Public Services International (PSI)",
+        "RAPAL",
+        "Rapal Uruguay",
+        "Recycling Partnership",
+        "Red De Acci\u00f3n Ecol\u00f3gica De M\u00e9xico",
+        "Red De Acci\u00f3n Por Los Derechos Ambientales",
+        "Red Mexicana De Accion Ecologica (Accion Ecologica)",
+        "Regions4 Sustainable Development",
+        "Reloop Platform",
+        "Resolve",
+        "Royal Society Of Chemistry",
+        "Samo Foundation",
+        "Sanid Organization For Relief And Development (SORD)",
+        "Sasakawa Peace Foundation",
+        "Saudi Green Building Forum",
+        "Sciaena",
+        "Scientists\u2019 Coalition For An Effective Plastics Treaty (Scientists\u2019 Coalition)",
+        "Secretariat For The Pacific Regional Environment Programme",
+        "Secretariat Of The Basel, Rotterdam And Stockholm Conventions",
+        "Secretariat Of The Convention On The Protection And Use Of Transboundary Watercourses And International Lakes (Water Convention)",
+        "Secretariat Of The Pacific Regional Environment Programme (SPREP)",
+        "Secretariat Of The WHO Framework Convention On Tobacco Control",
+        "Secretariats Of The Basel, Rotterdam And Stockholm Conventions",
+        "Shenzhen Zero Waste",
+        "Smoke Free Partnership, A Member Of The Stop Tobacco Pollution Alliance (STPA)",
+        "Sociedad Peruana De Derecho Ambiental (Peruvian Society Of Environmental Law)",
+        "Somali Sustainable Development Organisation (SOSDO)",
+        "Somali Youth Development Foundation (SYDF)",
+        "South Asia Cooperative Environment Programme",
+        "Stand Earth",
+        "Stevenson Holistic Care Foundation (SHCF)",
+        "Stichting CEFLEX \u2013 The Circular Economy For Flexible Packaging Initiative",
+        "Stiftelsen Stockholm International Water Institute",
+        "Styrenics Industry",
+        "Sustainable Coastlines Charitable Trust",
+        "Sustainable Environment Food And Agriculture Initiative",
+        "Swedish Society For Nature Conservation (SSNC)",
+        "Systemiq",
+        "T/A Plastics SA",
+        "Take 3 For The Sea",
+        "Taller Ecologista",
+        "Tangaroa Blue Foundation",
+        "Tearfund",
+        "Thailand",
+        "The Australian Marine Conservation Society",
+        "The Center For Oceanic Awareness, Research, And Education (COARE)",
+        "The Descendants Project",
+        "The Fletcher School",
+        "The Global Organization For PHA (GO!PHA)",
+        "The Nature Conservancy",
+        "The Ocean Cleanup",
+        "The Pew Charitable Trusts",
+        "The Sea Cleaners",
+        "The Society Of Native Nations",
+        "The Terracycle Foundation",
+        "The Vinyl Institute",
+        "Toxics Link",
+        "Toxisphera, Mingas Por El Mar",
+        "Trade Unions Major Group",
+        "Trash Hero World",
+        "Trash4tokens NGO",
+        "Tufts University",
+        "U.S. Council For International Business (USCIB)",
+        "UN Women\u2019s Major Group",
+        "UNESCO Association - Guwahati",
+        "UNESCO Chair For Ocean Sustainability",
+        "Udisha",
+        "Unbutton Fashion",
+        "Unions Workers And Wastepickers",
+        "United Nations Association of Spain and the Government of Catalonia",
+        "United Nations Conference On Trade And Development (UNCTAD)",
+        "United Nations Development Programme (UNDP)",
+        "United Nations Economic Commission For Europe (UNECE)",
+        "United Nations Global Compact",
+        "United Nations Human Settlements Programme (UN-Habitat)",
+        "United Nations Industrial Development Organization (UNIDO)",
+        "United Nations Institute For Training And Research (UNITAR)",
+        "United Nations Office For Disaster Risk Reduction (UNDRR)",
+        "United Nations Office On Drugs And Crime (UNODC)",
+        "United States Council For Business (USCIB)",
+        "University Of Wollongong",
+        "Unplastify",
+        "Verra",
+        "Vital Strategies",
+        "WWF-Australia",
+        "Waste Free Oceans",
+        "Whole World",
+        "William Ruto",
+        "Women In Informal Employment Globalizing And Organizing (WIEGO)",
+        "Women Working Group",
+        "Wonjin Institute For Occupational And Environmental Health (WIOEH)",
+        "Workers And Trade Unions Major Group",
+        "Working Group On Marine Litter (WGML) Of Coordinating Body On The Seas Of East Asia (COBSEA)",
+        "World Against Single Use Plastic (WASUP)",
+        "World Business Council For Sustainable Development (WBCSD)",
+        "World Economic Forum And Global Plastic Action Partnership (GPAP)",
+        "World Health Organisation",
+        "World Health Organization, Including The Secretariat Of The WHO Framework Convention On Tobacco Control",
+        "World Plastics Council (WPC)",
+        "World Welfare Association",
+        "World Wide Fund For Nature (WWF)",
+        "Wrap",
+        "Youth Alive Uganda",
+        "Youth Focus Group",
+        "Yunus Environment Hub",
+        "Zero Waste Europe",
+        "Zoological Society Of London (ZSL)"
+    ]
+}

data/authors_taxonomy.json

@@ -0,0 +1,520 @@
+{
+    "Members": {
+        "Countries": [
+            "Algeria",
+            "Angola",
+            "Antigua And Barbuda",
+            "Argentina",
+            "Armenia",
+            "Australia",
+            "Azerbaijan",
+            "Bahrain",
+            "Bangladesh",
+            "Barbados",
+            "Benin",
+            "Bosnia And Herzegovina",
+            "Brazil",
+            "Burkina Faso",
+            "Cambodia",
+            "Cameroon",
+            "Canada",
+            "Chile",
+            "China",
+            "Colombia",
+            "Comoros",
+            "Cook Islands",
+            "Costa Rica",
+            "Cuba",
+            "Cote D'Ivoire",
+            "Democratic Republic Of The Congo",
+            "Dominican Republic",
+            "Ecuador",
+            "Egypt",
+            "El Salvador",
+            "Equatorial Guinea",
+            "Eritrea",
+            "Eswatini",
+            "Ethiopia",
+            "Fiji",
+            "France",
+            "Gabon",
+            "Gambia",
+            "Georgia",
+            "Germany",
+            "Ghana",
+            "Grenada",
+            "Guatemala",
+            "Guinea",
+            "Guinea Bissau",
+            "Honduras",
+            "Iceland",
+            "India",
+            "Indonesia",
+            "Iraq",
+            "Islamic Republic Of Iran",
+            "Israel",
+            "Italy",
+            "Jamaica",
+            "Japan",
+            "Jordan",
+            "Kazakhstan",
+            "Kenya",
+            "Kiribati",
+            "Kuwait",
+            "Libya",
+            "Madagascar",
+            "Malawi",
+            "Malaysia",
+            "Maldives",
+            "Mali",
+            "Marshall Islands",
+            "Mauritius",
+            "Mexico",
+            "Micronesia",
+            "Moldova",
+            "Monaco",
+            "Mongolia",
+            "Montenegro",
+            "Morocco",
+            "Nauru",
+            "Nepal",
+            "New Zealand",
+            "Niger",
+            "Nigeria",
+            "North Macedonia",
+            "Norway",
+            "Oman",
+            "Pakistan",
+            "Palau",
+            "Panama",
+            "Papua New Guinea",
+            "Paraguay",
+            "Peru",
+            "Philippines",
+            "Qatar",
+            "Republic Of Congo",
+            "Republic Of Korea",
+            "Republic Of Moldova",
+            "Romania",
+            "Russian Federation",
+            "Rwanda",
+            "Samoa",
+            "Saudi Arabia",
+            "Senegal",
+            "Seychelles",
+            "Sierra Leone",
+            "Singapore",
+            "Solomon Islands",
+            "Somalia",
+            "South Africa",
+            "Sri Lanka",
+            "State Of Palestine",
+            "Switzerland",
+            "Syrian Arab Republic",
+            "Thailand",
+            "Timor-Leste",
+            "Togo",
+            "Tonga",
+            "Trinidad And Tobago",
+            "Tunisia",
+            "Turkey",
+            "Tuvalu",
+            "Uganda",
+            "Ukraine",
+            "United Arab Emirates",
+            "United Kingdom Of Great Britain And Northern Ireland",
+            "United Republic Of Tanzania",
+            "United States Of America",
+            "Uruguay",
+            "Venezuela",
+            "Vietnam",
+            "Yemen",
+            "Zambia",
+            "Zimbabwe"
+        ],
+        "International and Regional State Associations": [
+            "Asia-Pacific States",
+            "European Union (EU) And Its 27 Member States",
+            "Federated States Of Micronesia",
+            "Gulf Cooperation Council",
+            "High Ambition Coalition",
+            "Latin American And Caribbean Group (GRULAC)",
+            "Like Minded Countries",
+            "Pacific Small Island Developing States (PSIDS)",
+            "Alliance Of Small Island States (AOSIS)",
+            "The Federated States Of Micronesia",
+            "The Group Of African States",
+            "Western European And Other States",
+            "Coordinating Body on the Seas of East Asia (COBSEA)"
+        ]
+    },
+    "Intergovernmental Negotiation Committee": [
+        "Executive Secretary Of The INC",
+        "Intergovernmental Negotiation Committee"
+    ],
+    "Observers and Other Participants": [
+        "10YFP Secretariat (One Planet Network)",
+        "ABIPLAST - Brazilian Association Of The Plastic Industry",
+        "ACTS Organization",
+        "ASTM International (ASTM)",
+        "AWTAD Anti-Corruption Organization",
+        "Aaina",
+        "Action Et Education Pour Tous (AEPT)",
+        "Action On Smoking And Health (ASH)",
+        "Africa Climate And Environment Foundation (ACEF)",
+        "African Alliance For Health Research Economic Development",
+        "African Environmental Network",
+        "African Petroleum Producers\u2019 Organization (APPO)",
+        "Aguaclara",
+        "Alianza Basura Cero",
+        "Alianza Latinoamericana De Asociaciones De La Industria De Alimentos Y Bebidas (ALAIAB)",
+        "All-China Environment Federation",
+        "Alliance Panafricaine Multi-Acteurs Sur La Pollution Plastique",
+        "Alliance Pour Le Controle Du Tabac En Afrique (ACTA)",
+        "American Chemistry Council",
+        "American Fuel and Petrochemical Manufacturers (AFPM)",
+        "And Colectivo Jaguares De Nuestra Madre Tierra",
+        "Aotearoa Plastic Pollution Alliance (APPA)",
+        "Arctic Monitoring And Assessment Programme (AMAP)",
+        "Armenian Women For Health And Healthy Environment",
+        "Arnika",
+        "Asia Indigenous Peoples Pact And Indigenous Peoples",
+        "Asian Marine Conservation Association",
+        "Asociaci\u00f3n Ecol\u00f3gica Santo Tom\u00e1s",
+        "Asociaci\u00f3n Sustentar",
+        "Association For Promoting Sustainability In Campuses And Communities (APSCC)",
+        "Association For Rural Area Social Modification, Improvement And Nestling (ARASMIN)",
+        "Association For Supporting The SDGs For The United Nations",
+        "Association Institute Of Total Environment (INTEV)",
+        "Association Nationale Du Civisme",
+        "Association Of Leadership You-Lean",
+        "Association Of Plastic Recyclers",
+        "Association Of Solidarity Through Humanitarian Imperative Actions International (ASHIA)",
+        "Association Pour L'Integration Et La Developpement Durable Au Burundi",
+        "Australian Packaging Covenant Organisation (APCO)",
+        "Azul",
+        "BAN Toxics",
+        "BVRio",
+        "Barranquilla + 20",
+        "Basel Action Network (BAN)",
+        "Beijing Greenovation Institute For Public Welfare Development (GHub)",
+        "Brazilian Chemical Industry Association",
+        "Break Free From Plastic",
+        "Breathe Free Detroit",
+        "Bridgers Association Cameroon",
+        "Bureau Of International Recycling",
+        "Business And Industry Major Group Presented By World Business Council For Sustainable Development",
+        "Business Coalition For A Global Plastics Treaty Convened By The Ellen MacArthur Foundation And WWF, In Collaboration With Aligned Businesses And Financial Institutions, And Supported By NGO Partners",
+        "CAIRPLAS, C\u00e1mara Argentina De La Industria De Reciclados Plasticos",
+        "CDP",
+        "CEMPRE Colombia \u2013 Compromiso Empresarial Para El Reciclaje",
+        "CESTA",
+        "Carrizo Comecrudo Tribe",
+        "Center For Biological Diversity",
+        "Center For Coalfield Justice",
+        "Center For International Environmental Law (CIEL)",
+        "Center For International Law And Governance",
+        "Center For Islamic Studies Of Universitas Nasional, Jakarta",
+        "Center For Oceanic Awareness, Research, And Education (COARE)",
+        "Center For Public Health And Environmental Development (CEPHED)",
+        "Centre De Recherche Et D\u2019Education Pour Le D\u00e9veloppement (CREPD)",
+        "Centre D\u2019Accompagnement Des Alternatives Locales De D\u00e9veloppement (Caald)",
+        "Centre For Environmental Justice",
+        "Centre For Human Rights And Climate Change Research",
+        "Centre For Science And Environment (CSE)",
+        "Centre International De Droit Compar\u00e9 De L\u2019environnement",
+        "Centre International De Droit Compar\u00e9 De L\u2019environnement",
+        "Chemical And Allied Industries\u2019 Association (CAIA)",
+        "Chia Funkuin Foundation",
+        "Children And Youth Major Group (CYMG)",
+        "Children\u2019s Environmental Health Foundation",
+        "China Biodiversity Conservation And Green Development Foundation (CBCGDF)",
+        "Circular Economy For Flexible Packaging In Europe Initiative (CEFLEX)",
+        "Citeo",
+        "Citizen Consumer And Civic Action Group (CAG)",
+        "Civil Society And Rights Holder Coalition",
+        "Civil Society Organizations In Africa",
+        "Civil Society Organizations In Asia Pacific",
+        "Civil Society Organizations In Latin America",
+        "Co-habiter",
+        "Comision Centro Americana",
+        "Comit\u00e9 National Contre Le Tabagisme",
+        "Community Action Against Plastic Waste (CAPws)",
+        "Congregations Of St Joseph",
+        "Consciente Colectivo",
+        "Consumer Goods Forum",
+        "Consumers International",
+        "Contact Group 1",
+        "Contact Group 2",
+        "Convention On Biological Diversity (CBD)",
+        "Corporate Accountability",
+        "Council For Scientific And Industrial Research",
+        "C\u00e1mara De La Industria Qu\u00edmica Y Petroqu\u00edmica (CIQyP\u00ae), Miembro Del Concejo Internacional De Asociaciones Qu\u00edmicas (ICCA) Y EURECA",
+        "Danimer Scientific",
+        "EDANA And INDA",
+        "EPS Branchen-en Del Af Plastindustrien (Denmark)",
+        "EPS Industry Alliance",
+        "EURECA",
+        "Earth Day",
+        "Earth Law Center",
+        "Ecoplas",
+        "Ecoproject Partnership",
+        "Ellen MacArthur Foundation",
+        "Endocrine Society",
+        "Engineers Australia",
+        "Entidad Especializada En Pl\u00e1sticos Y Medio Ambiente Para Una Econom\u00eda Circular (ECOPLAS)",
+        "Entidades Unidas Reafirmando La Econom\u00eda Circular En Argentina (EURECA)",
+        "Environmental And Social Development Organization (ESDO)",
+        "Environmental Coalition On Standards (ECOS)",
+        "Environmental Development Association (FASEEL)",
+        "Environmental Investigation Agency",
+        "European Bioplastics (EUBP)",
+        "European Manufacturers Of EPS",
+        "European Manufacturers Of Expanded Polystyrene (EUMEPS)",
+        "Expanded Polystyrene Australia",
+        "Fauna and Flora International",
+        "Fauna and Flora International And Zoological Society Of London (ZSL)",
+        "Fenceline Watch",
+        "First Modern Agro Tools Common Initiative Group ( FI.MO.AT.C.I.G)",
+        "Fondation De La Mer",
+        "Food And Agriculture Organization Of The United Nations (FAO)",
+        "Food And Livestock Initiative (FLI Asbl)",
+        "Forum On Trade, Environment and The SDGs (TESS)",
+        "Foundation Of Fokus Nexus Tiga (Nexus3 Foundation)",
+        "French Water Partnership (Partenariat Fran\u00e7ais Pour L\u2019eau)",
+        "Friends World Committee For Consultation (FWCC)",
+        "Fronteras Comunes",
+        "Fundacion Avina",
+        "Fundaci\u00f3n Ambiente Y Recursos Naturales",
+        "Fundaci\u00f3n Interamericana Del Coraz\u00f3n (FIC)",
+        "GRID-Arendal",
+        "Galapagos Conservation Trust",
+        "Gallifrey Foundation",
+        "Geneva Cities Hub (GCH)",
+        "Gerakan Indonesia Diet Kantong Plastik (GIDKP) - The Indonesia Plastic Bag Diet Movement",
+        "Global Alliance For Incinerator Alternatives (GAIA)",
+        "Global Alliance On Health And Pollution (GAHP)",
+        "Global Cement And Concrete Association",
+        "Global Council For Science And The Environment (GCSE)",
+        "Global Plastics Policy Centre, University Of Portsmouth",
+        "Global Youth Coalition On Plastic Pollution (GYCPP)",
+        "Green Africa Youth Organization",
+        "Greenpeace International",
+        "GroundWork South Africa (GroundWorkSA)",
+        "Haitelmex Foundation",
+        "Hasiru Dala In Collaboration With Eleven Other Civil Society Organizations",
+        "Health And Environment Justice Support (HEJSupport)",
+        "Health Care Without Harm (HCWH)",
+        "Healthy Hospitals Project - PHS",
+        "Human Rights Watch",
+        "ICLEI - Local Governments For Sustainability",
+        "India Institute For Critical Action Centre In Movement (CACIM)",
+        "India Water Foundation",
+        "India Youth For Society",
+        "Indigenous Caucus",
+        "Indigenous Peoples And Their Communities Major Group",
+        "Indigenous Peoples Representatives",
+        "Indonesian Centre For Environmental Law (ICEL)",
+        "Innovazing Vision",
+        "Institute For Sustainable Development And Research (ISDR)",
+        "Integrated Strategies Forum",
+        "Interamerican Heart Foundation",
+        "International Air Transport Association",
+        "International Alliance Of Waste Pickers (IAWP)",
+        "International Alliance Of Waste-pickers",
+        "International Atomic Energy Agency (IAEA)",
+        "International Center Of Comparative Environmental Law (CIDCE)",
+        "International Centre For Environmental Education And Community Development (ICENECDEV)",
+        "International Chamber Of Commerce (ICC)",
+        "International Council Of Beverages Associations (ICBA)",
+        "International Council Of Chemical Associations (ICCA)",
+        "International Knowledge Hub Against Plastic Pollution (IKHAPP)",
+        "International Labour Office",
+        "International Labour Organization (ILO)",
+        "International Maritime Organization (IMO)",
+        "International Medical Crisis Response Alliance (IMCRA)",
+        "International Movement For Advancement Of Education Culture Social and Economic Development (IMAESED)",
+        "International Network For Bamboo And Rattan (INBAR)",
+        "International Organization For Standardization (ISO)",
+        "International Pollutants Elimination Network (IPEN)",
+        "International Science Council (ISC)",
+        "International Society Of Doctors For The Environment (ISDE)",
+        "International Solid Waste Association (ISWA)",
+        "International Trade Union Confederation (ITUC)",
+        "International Union For Conservation Of Nature And Natural Resources (IUCN)",
+        "Inuit Circumpolar Council (ICC)",
+        "Japan Clean Ocean Material Alliance (CLOMA)",
+        "King Abdullah Petroleum Studies And Research Center (KAPSARC)",
+        "Krityanand UNESCO Club, Jamshedpur",
+        "La Grande Puissance De Dieu",
+        "Latin American Organizations - Alianza Basura Cero",
+        "Ligue Camerounaise Des Droits De L'Homme",
+        "Litter4tokens South Africa NPO",
+        "Local And Subnational Government Working Group",
+        "Local Authorities",
+        "Loop",
+        "Major Alliance Education Centre (MAEC)",
+        "Major Group For Children And Youth",
+        "MarViva",
+        "Marine Ecosystems Protected Areas (MEPA) Trust",
+        "Members Of Microplastics Working Group",
+        "Mexican Network Of Ecological Action",
+        "Minderoo Foundation",
+        "Ministry Of Environment And Wildlife - Southwest State Of Somalia",
+        "Moms Clean Air Force",
+        "Multifaith Action Group On Pollution",
+        "NGO Major Group",
+        "NORCE On Behalf Of The North Atlantic Microplastic Centre (NAMC)",
+        "National Old Folks Of Liberia (NOFOL)",
+        "National Retail Association (NRA)",
+        "Natural Resources Defense Council (NRDC)",
+        "Neste",
+        "Nexus For Health, Environment, And Development (Nexus3) Foundation",
+        "Nipe Fagio",
+        "No Balloon Release Australia",
+        "No More Butts",
+        "Norwegian Academy Of International Law (NAIL)",
+        "Norwegian Institute For Water Research",
+        "Norwegian Institute For Water Research (NIVA)",
+        "Norwegian Research Centre (NORCE)",
+        "ONG Jeunesse Active De Guin\u00e9e (JAG)",
+        "Occidental Arts And Ecology Center (OAEC)",
+        "Ocean Conservancy",
+        "Ocean Recovery Alliance",
+        "Ocean. Now",
+        "OceanCare",
+        "OceanCare Global Ghost Gear Initiative",
+        "Office Of The UN High Commissioner For Human Rights (OHCHR)",
+        "OpenOceans Global",
+        "Organisation For Economic Co-operation And Development (OECD)",
+        "Organization Of Arab Petroleum Exporting Countries (OAPEC)",
+        "Organization Of The Petroleum Exporting Countries (OPEC)",
+        "Our Sea Of East Asia Network (OSEAN)",
+        "Out For Sustainability",
+        "PCX Solutions (HOPEx Environment Group, Inc)",
+        "Pacific Environment And Resources Center (Pacific Environment)",
+        "Pan American Neuroendocrine Society",
+        "Partnerships For Change",
+        "Paryavaran Mitra",
+        "PetStar",
+        "Planeteer Alliance And Captain Planet Foundation",
+        "Plastalliance - Alliance Plasturgie Et Composites Du Futur",
+        "Plastic Change",
+        "Plastic Free Foundation",
+        "Plastic Free Future",
+        "Plastic Oceans Australasia",
+        "Plastic Pollution Coalition",
+        "Plastics Federation Of South Africa",
+        "Plastics Industry Association",
+        "PlasticsEurope",
+        "Plasticulture",
+        "ProDelphinus",
+        "Public Services International (PSI)",
+        "RAPAL",
+        "Rapal Uruguay",
+        "Recycling Partnership",
+        "Red De Acci\u00f3n Ecol\u00f3gica De M\u00e9xico",
+        "Red De Acci\u00f3n Por Los Derechos Ambientales",
+        "Red Mexicana De Accion Ecologica (Accion Ecologica)",
+        "Regions4 Sustainable Development",
+        "Reloop Platform",
+        "Resolve",
+        "Royal Society Of Chemistry",
+        "Samo Foundation",
+        "Sanid Organization For Relief And Development (SORD)",
+        "Sasakawa Peace Foundation",
+        "Saudi Green Building Forum",
+        "Sciaena",
+        "Scientists\u2019 Coalition For An Effective Plastics Treaty (Scientists\u2019 Coalition)",
+        "Secretariat For The Pacific Regional Environment Programme",
+        "Secretariat Of The Basel, Rotterdam And Stockholm Conventions",
+        "Secretariat Of The Convention On The Protection And Use Of Transboundary Watercourses And International Lakes (Water Convention)",
+        "Secretariat Of The Pacific Regional Environment Programme (SPREP)",
+        "Secretariat Of The WHO Framework Convention On Tobacco Control",
+        "Secretariats Of The Basel, Rotterdam And Stockholm Conventions",
+        "Shenzhen Zero Waste",
+        "Smoke Free Partnership, A Member Of The Stop Tobacco Pollution Alliance (STPA)",
+        "Sociedad Peruana De Derecho Ambiental (Peruvian Society Of Environmental Law)",
+        "Somali Sustainable Development Organisation (SOSDO)",
+        "Somali Youth Development Foundation (SYDF)",
+        "South Asia Cooperative Environment Programme",
+        "Stand Earth",
+        "Stevenson Holistic Care Foundation (SHCF)",
+        "Stichting CEFLEX \u2013 The Circular Economy For Flexible Packaging Initiative",
+        "Stiftelsen Stockholm International Water Institute",
+        "Styrenics Industry",
+        "Sustainable Coastlines Charitable Trust",
+        "Sustainable Environment Food And Agriculture Initiative",
+        "Swedish Society For Nature Conservation (SSNC)",
+        "Systemiq",
+        "T/A Plastics SA",
+        "Take 3 For The Sea",
+        "Taller Ecologista",
+        "Tangaroa Blue Foundation",
+        "Tearfund",
+        "Thailand",
+        "The Australian Marine Conservation Society",
+        "The Center For Oceanic Awareness, Research, And Education (COARE)",
+        "The Descendants Project",
+        "The Fletcher School",
+        "The Global Organization For PHA (GO!PHA)",
+        "The Nature Conservancy",
+        "The Ocean Cleanup",
+        "The Pew Charitable Trusts",
+        "The Sea Cleaners",
+        "The Society Of Native Nations",
+        "The Terracycle Foundation",
+        "The Vinyl Institute",
+        "Toxics Link",
+        "Toxisphera, Mingas Por El Mar",
+        "Trade Unions Major Group",
+        "Trash Hero World",
+        "Trash4tokens NGO",
+        "Tufts University",
+        "U.S. Council For International Business (USCIB)",
+        "UN Women\u2019s Major Group",
+        "UNESCO Association - Guwahati",
+        "UNESCO Chair For Ocean Sustainability",
+        "Udisha",
+        "Unbutton Fashion",
+        "Unions Workers And Wastepickers",
+        "United Nations Association of Spain and the Government of Catalonia",
+        "United Nations Conference On Trade And Development (UNCTAD)",
+        "United Nations Development Programme (UNDP)",
+        "United Nations Economic Commission For Europe (UNECE)",
+        "United Nations Global Compact",
+        "United Nations Human Settlements Programme (UN-Habitat)",
+        "United Nations Industrial Development Organization (UNIDO)",
+        "United Nations Institute For Training And Research (UNITAR)",
+        "United Nations Office For Disaster Risk Reduction (UNDRR)",
+        "United Nations Office On Drugs And Crime (UNODC)",
+        "United States Council For Business (USCIB)",
+        "University Of Wollongong",
+        "Unplastify",
+        "Verra",
+        "Vital Strategies",
+        "WWF-Australia",
+        "Waste Free Oceans",
+        "Whole World",
+        "William Ruto",
+        "Women In Informal Employment Globalizing And Organizing (WIEGO)",
+        "Women Working Group",
+        "Wonjin Institute For Occupational And Environmental Health (WIOEH)",
+        "Workers And Trade Unions Major Group",
+        "Working Group On Marine Litter (WGML) Of Coordinating Body On The Seas Of East Asia (COBSEA)",
+        "World Against Single Use Plastic (WASUP)",
+        "World Business Council For Sustainable Development (WBCSD)",
+        "World Economic Forum And Global Plastic Action Partnership (GPAP)",
+        "World Health Organisation",
+        "World Health Organization, Including The Secretariat Of The WHO Framework Convention On Tobacco Control",
+        "World Plastics Council (WPC)",
+        "World Welfare Association",
+        "World Wide Fund For Nature (WWF)",
+        "Wrap",
+        "Youth Alive Uganda",
+        "Youth Focus Group",
+        "Yunus Environment Hub",
+        "Zero Waste Europe",
+        "Zoological Society Of London (ZSL)"
+    ]
+}

data/data wrangling/readme.txt

@@ -0,0 +1,2 @@
+data validation: check data for inconsistencies and errors
+data wrangling: update dataset

data/draftcat_taxonomy.json

@@ -0,0 +1,55 @@
+{ "part i": {                  "i": "part i: gerneral",
+                            "i1": "preamble",
+                            "i2": "objective",
+                            "i3": "definition",
+                            "i4": "principles",
+                            "i5": "scope"},
+                  "part ii": { "ii": "part ii: gerneral",
+                            "ii1": "primary plastic polymers",
+                            "ii2": "chemicals and polymers of concern",
+                            "ii3": "problematic and avoidable plastic products, including short-lived and single-use plastic products and intentionally added microplastics",
+                            "ii3a": "problematic and avoidable plastic products, including short-lived and single-use plastic products",
+                            "ii3b": "intentionally added microplastics",
+                            "ii4": "exemptions available to a Party upon request",
+                            "ii4bis":"dedicated programmes of work",
+                            "ii5": "product design, composition and performance",
+                            "ii5a": "product design and performance",
+                            "ii5b": "reduce, reuse, refill and repair of plastics and plastic products",
+                            "ii5c": "use of recycled plastic contents",
+                            "ii5d": "alternative plastics and plastic products",
+                            "ii6": "non-plastic substitutes",
+                            "ii7": "extended producer responsibility",
+                            "ii8": "emissions and releases of plastic throughout its life cycle",
+                            "ii9": "waste management",
+                            "ii9a":"waste management",
+                            "ii9b":"fishing gear",
+                            "ii10": "trade in listed chemicals, polymers and products, and in plastic waste",
+                            "ii10a": "trade in listed chemicals, polymers and products",
+                            "ii10b": "transboundary movement of plastic waste",
+                            "ii11": "existing plastic pollution, including in the marine environment",
+                            "ii12": "just transition",
+                            "ii13": "transparency, tracking, monitoring and labeling",
+                            "ii13bis": "overarching provision related to part ii"},
+                 "part iii": { "iii": "iii gerneral",
+                            "iii1": "financing",
+                             "iii2": "capacity-building, technical assistance and technology transfer"},
+                 "part iv": { "iv": "part iv: gerneral",
+                            "iv1": "national plans",
+                            "iv2": "implementation and compliance",
+                            "iv3": "reporting on progress",
+                            "iv4": "periodic assessment and monitoring of the progress of implementation of the instrument* and effectiveness evaluation",
+                            "iv4a": "effectiveness evaluation",
+                            "iv4b": "review of chemicals and polymers of concern, microplastics and problematic and avoidable products",
+                            "iv5": "international cooperation",
+                            "iv6": "information exchange",
+                            "iv7": "awareness-raising, education and research",
+                            "iv8": "stakeholder engagement",
+                            "iv8bis":"health aspects"},
+                  "part v": {"v": "part v: gerneral",
+                            "v0": "institutional arrangements",
+                            "v1": "governing body",
+                            "v2": "subsidiary bodies",
+                            "v3": "secretariat"},     
+                 "part vi": {"vi": "part vi: gerneral",
+                    "vi1": "final provisions"}
+                }

data/draftcat_taxonomy_filter.json

@@ -0,0 +1,64 @@
+{
+    "Part I": [ "part i: general",
+        "preamble",
+        "objective",
+        "definition",
+        "principles",
+        "scope"
+    ],
+    "Part II": ["part ii: general",
+        "primary plastic polymers",
+        "chemicals and polymers of concern",
+        "problematic and avoidable plastic products, including short-lived and single-use plastic products and intentionally added microplastics",
+        "problematic and avoidable plastic products, including short-lived and single-use plastic products",
+        "intentionally added microplastics",
+        "micro- and nanoplastics",
+        "exemptions available to a party upon request",
+        "dedicated programmes of work",
+        "product design, composition and performance",
+        "product design and performance",
+        "reduce, reuse, refill and repair of plastics and plastic products",
+        "use of recycled plastic contents",
+        "alternative plastics and plastic products",
+        "non-plastic substitutes",
+        "extended producer responsibility",
+        "emissions and releases of plastic throughout its life cycle",
+        "waste management",
+        "waste management",
+        "fishing gear",
+        "trade in listed chemicals, polymers and products, and in plastic waste",
+        "trade in listed chemicals, polymers and products",
+        "transboundary movement of plastic waste",
+        "existing plastic pollution, including in the marine environment",
+        "just transition",
+        "transparency, tracking, monitoring and labeling",
+        "overarching provision related to part ii"
+    ],
+    "Part III": ["part iii: general",
+        "financing",
+        "capacity-building, technical assistance and technology transfer"
+    ],
+    "Part IV": ["part iv: general",
+        "national plans",
+        "implementation and compliance",
+        "reporting on progress",
+        "periodic assessment and monitoring of the progress of implementation of the instrument* and effectiveness evaluation",
+        "assessment and monitoring",
+        "effectiveness evaluation",
+        "review of chemicals and polymers of concern, microplastics and problematic and avoidable products",
+        "international cooperation",
+        "information exchange",
+        "awareness-raising, education and research",
+        "stakeholder engagement",
+        "health aspects"
+    ],
+    "Part V": [ "part v: general",
+        "institutional arrangements",
+        "governing body",
+        "subsidiary bodies",
+        "secretariat"
+    ],
+    "Part VI": [ "part vi: general",
+        "final provisions"
+    ]
+}

data/example_prompts.json

@@ -0,0 +1,14 @@
+[
+    {
+        "id": 1,
+        "question": "What is Malysia's position on chemicals and polymers of concern?"
+    },
+    {
+        "id": 2,
+        "question": "Compare Indias and New Zealands position on just transition."
+    },
+    {
+        "id": 3,
+        "question": "Do the selected countries prefer a top-down instrument?"
+    }
+]

data/inc_df_v6_small_4.csv

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:388930deaa6033b41f3c6fb550f841c1bd1384a7ff86f4d0e775922f08617029
+size 28522667

data/taxonomies.txt

@@ -0,0 +1,497 @@
+authors_dict = {'Members': {'Countries': ['Algeria',
+  'Angola',
+  'Antigua And Barbuda',
+  'Argentina',
+  'Armenia',
+  'Australia',
+  'Azerbaijan',
+  'Bahrain',
+  'Bangladesh',
+  'Barbados',
+  'Benin',
+  'Bosnia And Herzegovina',
+  'Brazil',
+  'Burkina Faso',
+  'Cambodia',
+  'Cameroon',
+  'Canada',
+  'Chile',
+  'China',
+  'Colombia',
+  'Cook Islands',
+  'Costa Rica',
+  'Cuba',
+  'Democratic Republic Of The Congo',
+  'Dominican Republic',
+  'Ecuador',
+  'Egypt',
+  'El Salvador',
+  'Equatorial Guinea',
+  'Eritrea',
+  'Ethiopia',
+  'Fiji',
+  'France',
+  'Gabon',
+  'Georgia',
+  'Germany',
+  'Ghana',
+  'Grenada',
+  'Guatemala',
+  'Guinea',
+  'Guinea Bissau',
+  'Iceland',
+  'India',
+  'Indonesia',
+  'Islamic Republic Of Iran',
+  'Israel',
+  'Italy',
+  'Jamaica',
+  'Japan',
+  'Jordan',
+  'Kenya',
+  'Kiribati',
+  'Kuwait',
+  'Libya',
+  'Madagscar',
+  'Malawi',
+  'Malaysia',
+  'Mali',
+  'Marshall Islands',
+  'Mauritius',
+  'Mexico',
+  'Micronesia',
+  'Moldova',
+  'Monaco',
+  'Mongolia',
+  'Montenegro',
+  'Morocco',
+  'Nauru',
+  'Nepal',
+  'New Zealand',
+  'Niger',
+  'Nigeria',
+  'Norway',
+  'Oman',
+  'Pakistan',
+  'Palau',
+  'Panama',
+  'Papua New Guinea',
+  'Paraguay',
+  'Peru',
+  'Philippines',
+  'Qatar',
+  'Republic Of Congo',
+  'Republic Of Korea',
+  'Republic Of Moldova',
+  'Russian Federation',
+  'Rwanda',
+  'Samoa',
+  'Saudi Arabia',
+  'Senegal',
+  'Seychelles',
+  'Sierra Leone',
+  'Singapore',
+  'Somalia',
+  'South Africa',
+  'Sri Lanka',
+  'State Of Palestine',
+  'Switzerland',
+  'Syrian Arab Republic',
+  'Tonga',
+  'Trinidad And Tobago',
+  'Tunisia',
+  'Turkey',
+  'Uganda',
+  'Ukraine',
+  'United Arab Emirates',
+  'United Kingdom Of Great Britain And Northern Ireland',
+  'United Republic Of Tanzania',
+  'United States Of America',
+  'Uruguay',
+  'Venezuela',
+  'Vietnam',
+  'Yemen',
+  'Zambia'],
+ 'International and Regional State Associations': ['Alliance Of Small Island States (AOSIS)',
+  'Asia-Pacific States',
+  'European Union (EU) And Its 27 Member States',
+  'Federated States Of Micronesia',
+  'Gulf Cooperation Council',
+  'High Ambition Coalition',
+  'Latin American And Caribbean Group (GRULAC)',
+  'Pacific Small Island Developing States (PSIDS)',
+  'The Alliance Of Small Island States (AOSIS)',
+  'The Federated States Of Micronesia',
+  'The Group Of African States',
+  'Western European And Other States']
+},
+'Intergovernmenta Negotiation Committee': ['Executive Secretary Of The INC', 'Intergovernmental Negotiation Committee'],
+'Observers and Other Participants': [
+  '10YFP Secretariat (One Planet Network)',
+  'Aaina',
+  'ABIPLAST - Brazilian Association Of The Plastic Industry',
+  'Action Et Education Pour Tous (AEPT)',
+  'Action On Smoking And Health (ASH)',
+  'ACTS Organization',
+  'Africa Climate And Environment Foundation (ACEF)',
+  'African Alliance For Health Research Economic Development',
+  'African Environmental Network',
+  'African Petroleum Producers’ Organization (APPO)',
+  'Aguaclara',
+  'Alianza Basura Cero',
+  'Alianza Latinoamericana De Asociaciones De La Industria De Alimentos Y Bebidas (ALAIAB)',
+  'All-China Environment Federation',
+  'Alliance Pour Le Controle Du Tabac En Afrique (ACTA)',
+  'American Fuel and Petrochemical Manufacturers (AFPM)',
+  'And Colectivo Jaguares De Nuestra Madre Tierra',
+  'Aotearoa Plastic Pollution Alliance (APPA)',
+  'Arctic Monitoring And Assessment Programme (AMAP)',
+  'Arnika',
+  'Asian Marine Conservation Association',
+  'Asociación Ecológica Santo Tomás',
+  'Asociación Sustentar',
+  'Association For Promoting Sustainability In Campuses And Communities (APSCC)',
+  'Association For Rural Area Social Modification, Improvement And Nestling (ARASMIN)',
+  'Association For Supporting The SDGs For The United Nations',
+  'Association Institute Of Total Environment (INTEV)',
+  'Association Nationale Du Civisme',
+  'Association Of Leadership You-Lean',
+  'Association Of Plastic Recyclers',
+  'Association Of Solidarity Through Humanitarian Imperative Actions International (ASHIA)',
+  'ASTM International (ASTM)',
+  'Australian Packaging Covenant Organisation (APCO)',
+  'AWTAD Anti-Corruption Organization',
+  'Azul',
+  'BAN Toxics',
+  'Barranquilla + 20',
+  'Basel Action Network (BAN)',
+  'Beijing Greenovation Institute For Public Welfare Development (GHub)',
+  'Brazilian Chemical Industry Association',
+  'Breathe Free Detroit',
+  'Bridgers Association Cameroon',
+  'Bureau Of International Recycling',
+  'Business And Industry Major Group Presented By World Business Council For Sustainable Development',
+  'Business Coalition For A Global Plastics Treaty Convened By The Ellen MacArthur Foundation And WWF, In Collaboration With Aligned Businesses And Financial Institutions, And Supported By NGO Partners',
+  'BVRio',
+  'CAIRPLAS, Cámara Argentina De La Industria De Reciclados Plasticos',
+  'Cámara De La Industria Química Y Petroquímica (CIQyP®), Miembro Del Concejo Internacional De Asociaciones Químicas (ICCA) Y EURECA',
+  'Carrizo Comecrudo Tribe',
+  'CDP',
+  'CEMPRE Colombia – Compromiso Empresarial Para El Reciclaje',
+  'Center For Biological Diversity',
+  'Center For International Environmental Law (CIEL)',
+  'Center For International Law And Governance',
+  'Center For Islamic Studies Of Universitas Nasional, Jakarta',
+  'Center For Oceanic Awareness, Research, And Education (COARE)',
+  'Center For Public Health And Environmental Development (CEPHED)',
+  'Centre D’Accompagnement Des Alternatives Locales De Développement (Caald)',
+  'Centre De Recherche Et D’Education Pour Le Développement (CREPD)',
+  'Centre For Environmental Justice',
+  'Centre For Human Rights And Climate Change Research',
+  'Centre For Science And Environment (CSE)',
+  'Centre International De Droit Comparé De L’environnement',
+  'CESTA',
+  'Chemical And Allied Industries’ Association (CAIA)',
+  'Chia Funkuin Foundation',
+  'Children And Youth Major Group (CYMG)',
+  'Children’s Environmental Health Foundation',
+  'China Biodiversity Conservation And Green Development Foundation (CBCGDF)',
+  'Circular Economy For Flexible Packaging In Europe Initiative (CEFLEX)',
+  'Citeo',
+  'Citizen Consumer And Civic Action Group (CAG)',
+  'Civil Society And Rights Holder Coalition',
+  'Civil Society Organizations In Africa',
+  'Civil Society Organizations In Asia Pacific',
+  'Civil Society Organizations In Latin America',
+  'Co-habiter',
+  'Comité National Contre Le Tabagisme',
+  'Community Action Against Plastic Waste (CAPws)',
+  'Congregations Of St Joseph',
+  'Consciente Colectivo',
+  'Consumer Goods Forum',
+  'Consumers International',
+  'Contact Group 1',
+  'Contact Group 2',
+  'Convention On Biological Diversity (CBD)',
+  'Corporate Accountability',
+  'Council For Scientific And Industrial Research',
+  'Danimer Scientific',
+  'Earth Day',
+  'Earth Law Center',
+  'Ecoplas',
+  'Ecoproject Partnership',
+  'EDANA And INDA',
+  'Ellen MacArthur Foundation',
+  'Endocrine Society',
+  'Engineers Australia',
+  'Entidad Especializada En Plásticos Y Medio Ambiente Para Una Economía Circular (ECOPLAS)',
+  'Entidades Unidas Reafirmando La Economía Circular En Argentina (EURECA)',
+  'Environmental And Social Development Organization (ESDO)',
+  'Environmental Coalition On Standards (ECOS)',
+  'Environmental Development Association (FASEEL)',
+  'Environmental Investigation Agency',
+  'EPS Branchen-en Del Af Plastindustrien (Denmark)',
+  'EPS Industry Alliance',
+  'EURECA',
+  'European Bioplastics (EUBP)',
+  'European Manufacturers Of Expanded Polystyrene (EUMEPS)',
+  'Expanded Polystyrene Australia',
+  'Fauna and Flora International',
+  'Fenceline Watch',
+  'First Modern Agro Tools Common Initiative Group ( FI.MO.AT.C.I.G)',
+  'Food And Agriculture Organization Of The United Nations (FAO)',
+  'Food And Livestock Initiative (FLI Asbl)',
+  'Forum On Trade, Environment and The SDGs (TESS)',
+  'Foundation Of Fokus Nexus Tiga (Nexus3 Foundation)',
+  'French Water Partnership (Partenariat Français Pour L’eau)',
+  'Friends World Committee For Consultation (FWCC)',
+  'Fronteras Comunes',
+  'Fundación Ambiente Y Recursos Naturales',
+  'Fundacion Avina',
+  'Fundación Interamericana Del Corazón (FIC)',
+  'Galapagos Conservation Trust',
+  'Gallifrey Foundation',
+  'Geneva Cities Hub (GCH)',
+  'Gerakan Indonesia Diet Kantong Plastik (GIDKP) - The Indonesia Plastic Bag Diet Movement',
+  'Global Alliance For Incinerator Alternatives (GAIA)',
+  'Global Alliance On Health And Pollution (GAHP)',
+  'Global Cement And Concrete Association',
+  'Global Council For Science And The Environment (GCSE)',
+  'Global Plastics Policy Centre, University Of Portsmouth',
+  'Greenpeace International',
+  'GRID-Arendal',
+  'GroundWork South Africa (GroundWorkSA)',
+  'Haitelmex Foundation',
+  'Hasiru Dala In Collaboration With Eleven Other Civil Society Organizations',
+  'Health And Environment Justice Support (HEJSupport)',
+  'Health Care Without Harm (HCWH)',
+  'Healthy Hospitals Project - PHS',
+  'Human Rights Watch',
+  'ICLEI - Local Governments For Sustainability',
+  'India Institute For Critical Action Centre In Movement (CACIM)',
+  'India Water Foundation',
+  'Indigenous Peoples And Their Communities Major Group',
+  'Indigenous Peoples Representatives',
+  'Indonesian Centre For Environmental Law (ICEL)',
+  'Innovazing Vision',
+  'Institute For Sustainable Development And Research (ISDR)',
+  'Integrated Strategies Forum',
+  'Interamerican Heart Foundation',
+  'International Air Transport Association',
+  'International Alliance Of Waste-pickers',
+  'International Alliance Of Waste Pickers (IAWP)',
+  'International Atomic Energy Agency (IAEA)',
+  'International Center Of Comparative Environmental Law (CIDCE)',
+  'International Centre For Environmental Education And Community Development (ICENECDEV)',
+  'International Chamber Of Commerce (ICC)',
+  'International Council Of Beverages Associations (ICBA)',
+  'International Council Of Chemical Associations (ICCA)',
+  'International Knowledge Hub Against Plastic Pollution (IKHAPP)',
+  'International Labour Organization (ILO)',
+  'International Medical Crisis Response Alliance (IMCRA)',
+  'International Movement For Advancement Of Education Culture Social and Economic Development (IMAESED)',
+  'International Network For Bamboo And Rattan (INBAR)',
+  'International Organization For Standardization (ISO)',
+  'International Pollutants Elimination Network (IPEN)',
+  'International Science Council (ISC)',
+  'International Society Of Doctors For The Environment (ISDE)',
+  'International Solid Waste Association (ISWA)',
+  'International Trade Union Confederation (ITUC)',
+  'International Union For Conservation Of Nature And Natural Resources (IUCN)',
+  'Inuit Circumpolar Council (ICC)',
+  'Japan Clean Ocean Material Alliance (CLOMA)',
+  'King Abdullah Petroleum Studies And Research Center (KAPSARC)',
+  'Krityanand UNESCO Club, Jamshedpur',
+  'La Grande Puissance De Dieu',
+  'Latin American Organizations - Alianza Basura Cero',
+  'Litter4tokens South Africa NPO',
+  'Major Alliance Education Centre (MAEC)',
+  'Major Group For Children And Youth',
+  'Marine Ecosystems Protected Areas (MEPA) Trust',
+  'MarViva',
+  'Members Of Microplastics Working Group',
+  'Mexican Network Of Ecological Action',
+  'Minderoo Foundation',
+  'Ministry Of Environment And Wildlife - Southwest State Of Somalia',
+  'Moms Clean Air Force',
+  'Multifaith Action Group On Pollution',
+  'National Old Folks Of Liberia (NOFOL)',
+  'National Retail Association (NRA)',
+  'Natural Resources Defense Council (NRDC)',
+  'Neste',
+  'Nexus For Health, Environment, And Development (Nexus3) Foundation',
+  'NGO Major Group',
+  'Nipe Fagio',
+  'No Balloon Release Australia',
+  'No More Butts',
+  'NORCE On Behalf Of The North Atlantic Microplastic Centre (NAMC)',
+  'Norwegian Academy Of International Law (NAIL)',
+  'Norwegian Institute For Water Research (NIVA)',
+  'Norwegian Research Centre (NORCE)',
+  'Occidental Arts And Ecology Center (OAEC)',
+  'Ocean Conservancy',
+  'Ocean Recovery Alliance',
+  'Ocean. Now',
+  'OceanCare',
+  'Office Of The UN High Commissioner For Human Rights (OHCHR)',
+  'ONG Jeunesse Active De Guinée (JAG)',
+  'OpenOceans Global',
+  'Organisation For Economic Co-operation And Development (OECD)',
+  'Organization Of Arab Petroleum Exporting Countries (OAPEC)',
+  'Organization Of The Petroleum Exporting Countries (OPEC)',
+  'Our Sea Of East Asia Network (OSEAN)',
+  'Out For Sustainability',
+  'Pacific Environment And Resources Center (Pacific Environment)',
+  'Pan American Neuroendocrine Society',
+  'Partnerships For Change',
+  'Paryavaran Mitra',
+  'PCX Solutions (HOPEx Environment Group, Inc)',
+  'PetStar',
+  'Planeteer Alliance And Captain Planet Foundation',
+  'Plastalliance - Alliance Plasturgie Et Composites Du Futur',
+  'Plastic Change',
+  'Plastic Free Foundation',
+  'Plastic Free Future',
+  'Plastic Oceans Australasia',
+  'Plastic Pollution Coalition',
+  'Plastics Federation Of South Africa',
+  'Plastics Industry Association',
+  'Plasticulture',
+  'ProDelphinus',
+  'RAPAL',
+  'Recycling Partnership',
+  'Red De Acción Ecológica De México',
+  'Red De Acción Por Los Derechos Ambientales',
+  'Red Mexicana De Accion Ecologica (Accion Ecologica)',
+  'Regions4 Sustainable Development',
+  'Reloop Platform',
+  'Resolve',
+  'Samo Foundation',
+  'Sanid Organization For Relief And Development (SORD)',
+  'Sasakawa Peace Foundation',
+  'Sciaena',
+  'Scientists’ Coalition For An Effective Plastics Treaty (Scientists’ Coalition)',
+  'Secretariat For The Pacific Regional Environment Programme',
+  'Secretariat Of The Convention On The Protection And Use Of Transboundary Watercourses And International Lakes (Water Convention)',
+  'Secretariat Of The Pacific Regional Environment Programme (SPREP)',
+  'Secretariat Of The WHO Framework Convention On Tobacco Control',
+  'Secretariats Of The Basel, Rotterdam And Stockholm Conventions',
+  'Shenzhen Zero Waste',
+  'Smoke Free Partnership, A Member Of The Stop Tobacco Pollution Alliance (STPA)',
+  'Sociedad Peruana De Derecho Ambiental (Peruvian Society Of Environmental Law)',
+  'Somali Sustainable Development Organisation (SOSDO)',
+  'Somali Youth Development Foundation (SYDF)',
+  'Stevenson Holistic Care Foundation (SHCF)',
+  'Stichting CEFLEX – The Circular Economy For Flexible Packaging Initiative',
+  'Stiftelsen Stockholm International Water Institute',
+  'Styrenics Industry',
+  'Sustainable Coastlines Charitable Trust',
+  'Sustainable Environment Food And Agriculture Initiative',
+  'Swedish Society For Nature Conservation (SSNC)',
+  'T/A Plastics SA',
+  'Take 3 For The Sea',
+  'Taller Ecologista',
+  'Tangaroa Blue Foundation',
+  'Tearfund',
+  'Thailand',
+  'The Australian Marine Conservation Society',
+  'The Center For Oceanic Awareness, Research, And Education (COARE)',
+  'The Descendants Project',
+  'The Fletcher School',
+  'The Global Organization For PHA (GO!PHA)',
+  'The Ocean Cleanup',
+  'The Pew Charitable Trusts',
+  'The Sea Cleaners',
+  'The Society Of Native Nations',
+  'The Vinyl Institute',
+  'Toxisphera, Mingas Por El Mar',
+  'Trade Unions Major Group',
+  'Trash Hero World',
+  'Trash4tokens NGO',
+  'Tufts University',
+  'U.S. Council For International Business (USCIB)',
+  'Udisha',
+  'Unbutton Fashion',
+  'UNESCO Association - Guwahati',
+  'UNESCO Chair For Ocean Sustainability',
+  'Unions Workers And Wastepickers',
+  'United Nations Conference On Trade And Development (UNCTAD)',
+  'United Nations Development Programme (UNDP)',
+  'United Nations Economic Commission For Europe (UNECE)',
+  'United Nations Global Compact',
+  'United Nations Human Settlements Programme (UN-Habitat)',
+  'United Nations Industrial Development Organization (UNIDO)',
+  'United Nations Institute For Training And Research (UNITAR)',
+  'United Nations Office For Disaster Risk Reduction (UNDRR)',
+  'United Nations Office On Drugs And Crime (UNODC)',
+  'University Of Wollongong',
+  'Unplastify',
+  'Verra',
+  'Vital Strategies',
+  'Waste Free Oceans',
+  'Women In Informal Employment Globalizing And Organizing (WIEGO)',
+  'Wonjin Institute For Occupational And Environmental Health (WIOEH)',
+  'World Against Single Use Plastic (WASUP)',
+  'World Business Council For Sustainable Development (WBCSD)',
+  'World Economic Forum And Global Plastic Action Partnership (GPAP)',
+  'World Health Organisation',
+  'World Health Organization, Including The Secretariat Of The WHO Framework Convention On Tobacco Control',
+  'World Plastics Council (WPC)',
+  'World Wide Fund For Nature (WWF)',
+  'Wrap',
+  'WWF-Australia',
+  'Youth Focus Group',
+  'Yunus Environment Hub',
+  'Zero Waste Europe',
+  'Zoological Society Of London (ZSL)']}
+
+authors_dict
+
+draftcat_dict = { 'part i': {'i1': 'preamble',
+                            'i2': 'objective',
+                            'i3': 'definition',
+                            'i4': 'principles',
+                            'i5': 'scope'},
+                  'part ii': {'ii1': 'primary plastic polymers',
+                            'ii2': 'chemicals and polymers of concern',
+                            'ii3': 'problematic and avoidable plastic products, including short-lived and single-use plastic products and intentionally added microplastics',
+                            'ii3a': 'problematic and avoidable plastic products, including short-lived and single-use plastic products',
+                            'ii3b': 'intentionally added microplastics',
+                            'ii4': 'exemptions available to a Party upon request',
+                            'ii5': 'product design, composition and performance',
+                            'ii5a': 'product design and performance',
+                            'ii5b': 'reduce, reuse, refill and repair of plastics and plastic products',
+                            'ii5c': 'use of recycled plastic contents',
+                            'ii5d': 'alternative plastics and plastic products',
+                            'ii6': 'non-plastic substitutes',
+                            'ii7': 'extended producer responsibility',
+                            'ii8': 'emissions and releases of plastic throughout its life cycle',
+                            'ii9': 'waste management',
+                            'ii9a':'waste management',
+                            'ii9b':'fishing gear',
+                            'ii10': 'trade in listed chemicals, polymers and products, and in plastic waste',
+                            'ii10a': 'trade in listed chemicals, polymers and products',
+                            'ii10b': 'transboundary movement of plastic waste',
+                            'ii11': 'existing plastic pollution, including in the marine environment',
+                            'ii12': 'just transition',
+                            'ii13': 'transparency, tracking, monitoring and labeling'},
+                 'part iii': {'iii1': 'financing',
+                             'iii2': 'capacity-building, technical assistance and technology transfer'},
+                 'part iv': {'iv1': 'national plans',
+                            'iv2': 'implementation and compliance',
+                            'iv3': 'reporting on progress ',
+                            'iv4': 'periodic assessment and monitoring of the progress of implementation of the instrument* and effectiveness evaluation',
+                            'iv4a': 'effectiveness evaluation',
+                            'iv4b': 'review of chemicals and polymers of concern, microplastics and problematic and avoidable products',
+                            'iv5': 'international cooperation',
+                            'iv6': 'information exchange',
+                            'iv7': 'awareness-raising, education and research',
+                            'iv8': 'stakeholder engagement'},
+                  'part v': {'part v': 'institutional arrangements',
+                            'v1': 'governing body',
+                            'v2': 'subsidiary bodies',
+                            'v3': 'secretariat'},     
+                 'part vi': {'part vi': 'final provisions'}
+                }
+
+draftcat_dict

database/document_store.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b056059cccd69629ccbb81260f3bf0f8cfa3513d75c56fc402495cf1af2b5a3
+size 133

pyproject.toml

@@ -0,0 +1,30 @@
+[tool.poetry]
+name = "plastic-treaty-app"
+version = "0.1.0"
+description = ""
+authors = ["Rahkakavee Baskaran <[email protected]>"]
+readme = "README.md"
+packages = [{include = "plastic_treaty_app"}]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+streamlit = "^1.30.0"
+farm-haystack = "^1.23.0"
+pydantic = "<2"
+load-dotenv = "^0.1.0"
+torch = "^2.1.2"
+nltk = "^3.8.1"
+sentence-transformers = "^2.2.2"
+scikit-learn = "^1.4.1.post1"
+langchain = "^0.1.9"
+
+
+[tool.poetry.group.dev.dependencies]
+flake8 = "^7.0.0"
+pre-commit = "^3.6.0"
+mypy = "^1.8.0"
+black = "^24.1.1"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

src/app/v1/app.py

@@ -0,0 +1,329 @@
+import os
+import sys
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..')))
+import pandas as pd
+
+
+from src.rag.pipeline import RAGPipeline
+import streamlit as st
+from src.utils.data import (
+    build_filter,
+    get_filter_values,
+    get_meta,
+    load_json,
+    load_css,
+)
+from src.utils.writer import typewriter
+
+st.set_page_config(layout="wide")
+
+
+
+EMBEDDING_MODEL = "sentence-transformers/distiluse-base-multilingual-cased-v1"
+PROMPT_TEMPLATE = os.path.join("src", "rag", "prompt_template.yaml")
+
+
+@st.cache_data
+def load_css_style(path: str) -> None:
+    load_css(path)
+
+
+@st.cache_data
+def get_meta_data() -> pd.DataFrame:
+    return pd.read_csv(
+        os.path.join("database", "meta_data.csv"), dtype=({"retriever_id": str})
+    )
+
+
+@st.cache_data
+def get_authors_taxonomy() -> dict[str, list[str]]:
+    return load_json(os.path.join("data", "authors_filter.json"))
+
+
+@st.cache_data
+def get_draft_cat_taxonomy() -> dict[str, list[str]]:
+    return load_json(os.path.join("data", "draftcat_taxonomy_filter.json"))
+
+
+@st.cache_data
+def get_example_prompts() -> list[str]:
+    return [
+        example["question"]
+        for example in load_json(os.path.join("data", "example_prompts.json"))
+    ]
+
+
+@st.cache_resource
+def load_pipeline() -> RAGPipeline:
+    return RAGPipeline(
+        embedding_model=EMBEDDING_MODEL,
+        prompt_template=PROMPT_TEMPLATE,
+    )
+
+
+@st.cache_data
+def load_app_init() -> None:
+    # Define the title of the app
+    st.title("INC Plastic Treaty - Q&A")
+
+    # add warning emoji and style
+    st.markdown(
+        """
+        <p class="remark"> ⚠️ Remark:
+        The app is a beta version that serves as a basis for further development. We are aware that the performance is not yet sufficient and that the data basis is not yet complete. We are grateful for any feedback that contributes to the further development and improvement of the app!
+    """,
+        unsafe_allow_html=True,
+    )
+
+    # add explanation to the app
+    st.markdown(
+        """
+        <p class="description">
+        The app aims to facilitate the search for information and documents related to the UN Plastics Treaty Negotiations. The database includes all relevant documents that are available <a href=https://www.unep.org/inc-plastic-pollution target="_blank">here</a>. Users can query the data through a chatbot. Please note that, due to technical constraints, only a maximum of 10 documents can be used to generate the answer. A comprehensive response can therefore not be guaranteed. However, all relevant documents can be accessed via a link using the filter functions.
+        Filter functions are available to narrow down the data by country/author, zero draft categories and negotiation rounds. Pre-selecting relevant data enhances the accuracy of generated answers. Additionally, all documents selected via the filter function can be accessed via a link.
+        """,
+        unsafe_allow_html=True,
+    )
+
+
+load_css_style("style/style.css")
+
+
+# Load the data
+metadata = get_meta_data()
+authors_taxonomy = get_authors_taxonomy()
+draft_cat_taxonomy = get_draft_cat_taxonomy()
+example_prompts = get_example_prompts()
+
+# Load pipeline
+pipeline = load_pipeline()
+
+# Load app init
+load_app_init()
+
+
+filter_col = st.columns(1)
+# Filter column
+with filter_col[0]:
+    st.markdown("## Select Filters")
+    author_col, round_col, draft_cat_col = st.columns([1, 1, 1])
+
+    with author_col:
+        st.markdown("### Authors")
+        selected_author_parent = st.multiselect(
+            "Entity Parent", list(authors_taxonomy.keys())
+        )
+
+        available_child_items = []
+        for category in selected_author_parent:
+            available_child_items.extend(authors_taxonomy[category])
+
+        selected_authors = st.multiselect("Entity", available_child_items)
+
+    with round_col:
+        st.markdown("### Round")
+        negotiation_rounds = get_filter_values(metadata, "round")
+        selected_rounds = st.multiselect("Round", negotiation_rounds)
+
+    with draft_cat_col:
+        st.markdown("### Draft Categories")
+        selected_draft_cats_parent = st.multiselect(
+            "Draft Categories Parent", list(draft_cat_taxonomy.keys())
+        )
+        available_draft_cats_child_items = []
+        for category in selected_draft_cats_parent:
+            available_draft_cats_child_items.extend(draft_cat_taxonomy[category])
+
+        selected_draft_cats = st.multiselect(
+            "Draft Categories", available_draft_cats_child_items
+        )
+
+
+prompt_col, output_col = st.columns([1, 1.5])
+# make the buttons text smaller
+
+
+# GPT column
+with prompt_col:
+    st.markdown("## Filter documents")
+    st.markdown(
+        """
+        * The filter function allows you to see all documents that match the selected filters.
+        * Additionally, all documents selected via the filter function can be accessed via a link.
+        * Alternatively, you can ask a question to the model. The model will then provide you with an answer based on the filtered documents.
+        """
+    )
+    trigger_filter = st.session_state.setdefault("trigger", False)
+    if st.button("Filter documents"):
+        filter_selection_transformed = build_filter(
+            meta_data=metadata,
+            authors_filter=selected_authors,
+            draft_cats_filter=selected_draft_cats,
+            round_filter=selected_rounds,
+        )
+        documents = pipeline.document_store.get_all_documents(
+            filters=filter_selection_transformed
+        )
+        trigger_filter = True
+
+    st.markdown("## Ask a question")
+    if "prompt" not in st.session_state:
+        prompt = st.text_area("")
+    if (
+        "prompt" in st.session_state
+        and st.session_state.prompt in example_prompts  # noqa: E501
+    ):  # noqa: E501
+        prompt = st.text_area(
+            "Enter a question", value=st.session_state.prompt
+        )  # noqa: E501
+    if (
+        "prompt" in st.session_state
+        and st.session_state.prompt not in example_prompts  # noqa: E501
+    ):  # noqa: E501
+        del st.session_state["prompt"]
+        prompt = st.text_area("Enter a question")
+
+    trigger_ask = st.session_state.setdefault("trigger", False)
+    if st.button("Ask"):
+        with st.status("Filtering documents...", expanded=False) as status:
+            if filter_selection_transformed == {}:
+                st.warning(
+                    "No filters selected. We highly recommend to use filters otherwise the answer might not be accurate. In addition you might experience performance issues since the model has to analyze all available documents."
+                )
+            filter_selection_transformed = build_filter(
+                meta_data=metadata,
+                authors_filter=selected_authors,
+                draft_cats_filter=selected_draft_cats,
+                round_filter=selected_rounds,
+            )
+
+            documents = pipeline.document_store.get_all_documents(
+                filters=filter_selection_transformed
+            )
+            status.update(
+                label="Filtering documents completed!", state="complete", expanded=False
+            )
+        with st.status("Answering question...", expanded=True) as status:
+            result = pipeline(prompt=prompt, filters=filter_selection_transformed)
+            trigger_ask = True
+            status.update(
+                label="Answering question completed!", state="complete", expanded=False
+            )
+
+    st.markdown("### Examples")
+    st.markdown(
+        """
+        * These are example prompts that can be used to ask questions to the model
+        * Click on a prompt to use it as a question. You can also type your own question in the text area above.
+        * For questions like "How do country a, b and c [...]" please make sure to select the countries in the filter section. Otherwise the answer will not be accurate. In general we highly recommend to use the filter functions to narrow down the data.
+        """
+    )
+
+    for i, prompt in enumerate(example_prompts):
+        # with col[i % 4]:
+        if st.button(prompt):
+            if "key" not in st.session_state:
+                st.session_state["prompt"] = prompt
+    # Define the button
+
+
+if trigger_ask:
+    with output_col:
+        meta_data = get_meta(result=result)
+        answer = result["answers"][0].answer
+
+        meta_data_cleaned = []
+        seen_retriever_ids = set()
+
+        for data in meta_data:
+            retriever_id = data["retriever_id"]
+            content = data["content"]
+            if retriever_id not in seen_retriever_ids:
+                meta_data_cleaned.append(
+                    {
+                        "retriever_id": retriever_id,
+                        "href": data["href"],
+                        "content": [content],
+                    }
+                )
+                seen_retriever_ids.add(retriever_id)
+            else:
+                for i, item in enumerate(meta_data_cleaned):
+                    if item["retriever_id"] == retriever_id:
+                        meta_data_cleaned[i]["content"].append(content)
+
+        references = ["\n"]
+        for data in meta_data_cleaned:
+            retriever_id = data["retriever_id"]
+            href = data["href"]
+            references.append(f"-[{retriever_id}]: {href} \n")
+        st.write("#### 📌 Answer")
+        typewriter(
+            text=answer,
+            references=references,
+            speed=100,
+        )
+
+        with st.expander("Show more information to the documents"):
+            for data in meta_data_cleaned:
+                markdown_text = f"- Document: {data['retriever_id']}\n"
+                markdown_text += "    - Text passages\n"
+                for content in data["content"]:
+                    content = content.replace("[", "").replace("]", "").replace("'", "")
+                    content = " ".join(content.split())
+                    markdown_text += f"        - {content}\n"
+                st.write(markdown_text)
+
+    col4 = st.columns(1)
+    with col4[0]:
+        references = []
+        for document in documents:
+            authors = document.meta["author"]
+            authors = authors.replace("'", "").replace("[", "").replace("]", "")
+            href = document.meta["href"]
+            source = f"- {authors}: {href}"
+            references.append(source)
+            references = list(set(references))
+            references = sorted(references)
+        st.markdown("### Overview of all filtered documents")
+        st.markdown(
+            f"<p class='description'> The answer above results from the most similar text passages (top 7) from the documents that you can find under 'References' in the answer block. Below you will find an overview of all documents that match the filters you have selected. Please note that the above answer is based specifically on the highlighted references above and does not include the findings from all the filtered documents shown below. \n For your current filtering, {len(references)} documents were found. </p>",
+            unsafe_allow_html=True,
+        )
+        for reference in references:
+            st.write(reference)
+    trigger = 0
+
+
+if trigger_filter:
+    with output_col:
+        references = []
+        for document in documents:
+            authors = document.meta["author"]
+            authors = authors.replace("'", "").replace("[", "").replace("]", "")
+            href = document.meta["href"]
+            round_ = document.meta["round"]
+            draft_labs = document.meta["draft_labs"]
+            references.append(
+                {
+                    "author": authors,
+                    "href": href,
+                    "draft_labs": draft_labs,
+                    "round": round_,
+                }
+            )
+        references = pd.DataFrame(references)
+        references = references.drop_duplicates()
+        st.markdown("### Overview of all filtered documents")
+        # show
+        # make columns author and draft_labs bigger and make href width smaller and round width smaller
+        st.dataframe(
+            references,
+            hide_index=True,
+            column_config={
+                "author": st.column_config.ListColumn("Authors"),
+                "href": st.column_config.LinkColumn("Link to Document"),
+                "draft_labs": st.column_config.ListColumn("Draft Categories"),
+                "round": st.column_config.NumberColumn("Round"),
+            },
+        )

src/app/v2/app.py

@@ -0,0 +1,385 @@
+import os
+import sys
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..')))
+import pandas as pd
+import streamlit as st
+import time
+
+
+from src.rag.pipeline import RAGPipeline
+from src.utils.data_v2 import (
+    build_filter,
+    get_meta,
+    load_json,
+    load_css,
+)
+from src.utils.writer import typewriter
+
+
+st.set_page_config(layout="wide")
+
+EMBEDDING_MODEL = "sentence-transformers/distiluse-base-multilingual-cased-v1"
+PROMPT_TEMPLATE = os.path.join("src", "rag", "prompt_template.yaml")
+
+
+@st.cache_data
+def load_css_style(path: str) -> None:
+    load_css(path)
+
+
+@st.cache_data
+def get_meta_data() -> pd.DataFrame:
+    return pd.read_csv(
+        os.path.join("database", "meta_data.csv"), dtype=({"retriever_id": str})
+    )
+
+
+@st.cache_data
+def get_df() -> pd.DataFrame:
+    return pd.read_csv(
+        os.path.join("data", "inc_df.csv"), dtype=({"retriever_id": str})
+    )[["retriever_id", "draft_labs", "author", "href", "round"]]
+
+
+@st.cache_data
+def get_authors_taxonomy() -> list[str]:
+    taxonomy = load_json(os.path.join("data", "authors_taxonomy.json"))
+    countries = []
+    members = taxonomy["Members"]
+    for key, value in members.items():
+        if key == "Countries" or key == "International and Regional State Associations":
+            countries.extend(value)
+    return countries
+
+
+@st.cache_data
+def get_draft_cat_taxonomy() -> dict[str, list[str]]:
+    taxonomy = load_json(os.path.join("data", "draftcat_taxonomy_filter.json"))
+    draft_labels = []
+    for _, subpart in taxonomy.items():
+        for label in subpart:
+            draft_labels.append(label)
+    return draft_labels
+
+
+@st.cache_data
+def get_example_prompts() -> list[str]:
+    return [
+        example["question"]
+        for example in load_json(os.path.join("data", "example_prompts.json"))
+    ]
+
+
+@st.cache_data
+def set_trigger_state_values() -> tuple[bool, bool]:
+    trigger_filter = st.session_state.setdefault("trigger", False)
+    trigger_ask = st.session_state.setdefault("trigger", False)
+    return trigger_filter, trigger_ask
+
+
+@st.cache_resource
+def load_pipeline() -> RAGPipeline:
+    return RAGPipeline(
+        embedding_model=EMBEDDING_MODEL,
+        prompt_template=PROMPT_TEMPLATE,
+    )
+
+
+@st.cache_data
+def load_app_init() -> None:
+    # Define the title of the app
+    st.title("INC Plastic Pollution Country Profile Analysis")
+
+    # add warning emoji and style
+
+    st.markdown(
+        """
+    <div class="remark">
+        <div class="remark-content">
+            <p class="remark-text" style="font-size: 20px;"> ⚠️ The app is a beta version that serves as a basis for further development. We are aware that the performance is not yet sufficient and that the data basis is not yet complete. We are grateful for any feedback that contributes to the further development and improvement of the app!</p>
+        </div>
+    </div>
+    """,
+        unsafe_allow_html=True,
+    )
+
+    st.markdown(
+        """
+        <a href="mailto:[email protected]" class="feedback-link">Send feedback</a>
+        """,
+        unsafe_allow_html=True,
+    )
+
+    # add explanation to the app
+    st.markdown(
+        """
+        <p class="description">
+        The app is tailored to enhance the efficiency of finding and accessing information on the UN Plastics Treaty Negotiations. It hosts a comprehensive database of relevant documents submitted by the members available <a href="https://www.unep.org/inc-plastic-pollution"> here</a>, which users can explore through an intuitive chatbot interface as well as simple filtering options.
+        The app excels in querying specific information about countries and their positions in  the negotiations, providing targeted and precise answers. However, it can process only up to 8 relevant documents at a time, which may limit responses to more complex inquiries. Filter options by authors and sections of the negotiation draft ensure the accuracy of the answers. Each document found via these filters is also directly accessible via a link, ensuring complete and easy access to the desired information.
+        </p>
+        """,
+        unsafe_allow_html=True,
+    )
+
+
+load_css_style("style/style.css")
+
+
+# Load the data
+df = get_df()
+df_transformed = get_meta_data()
+countries = get_authors_taxonomy()
+draft_labels = get_draft_cat_taxonomy()
+example_prompts = get_example_prompts()
+trigger_filter, trigger_ask = set_trigger_state_values()
+
+# Load pipeline
+pipeline = load_pipeline()
+
+# Load app init
+load_app_init()
+
+
+application_col = st.columns(1)
+
+
+with application_col[0]:
+    st.markdown("""<p class="header"> 1️⃣ Select countries""", unsafe_allow_html=True)
+    st.markdown(
+        """
+        <p class="description">
+        Please select the countries of interest. Your selection will refine the database to include documents submitted by these countries or recognized groupings such as Small Developing States, the African States Group, etc. </p>
+        """,
+        unsafe_allow_html=True,
+    )
+    selected_authors = st.multiselect(
+        label="country",
+        options=countries,
+        label_visibility="collapsed",
+        placeholder="Select country/countries",
+    )
+
+    st.write("\n")
+    st.write("\n")
+
+    st.markdown(
+        """<p class="header">  2️⃣ Select parts of the negotiation draft""",
+        unsafe_allow_html=True,
+    )
+    st.markdown(
+        """
+        <p class="description">
+        Please select the parts of the negotiation draft of interest. The negotiation draft can be accessed <a href="https://www.unep.org/inc-plastic-pollution/session-4/documents"> here</a>. </p>
+        """,
+        unsafe_allow_html=True,
+    )
+    selected_draft_cats = st.multiselect(
+        label="Subpart",
+        options=draft_labels,
+        label_visibility="collapsed",
+        placeholder="Select draft category/draft categories",
+    )
+
+    st.write("\n")
+    st.write("\n")
+
+    st.markdown(
+        """<p class="header"> 3️⃣ Ask a question or show documents based on selected filters""",
+        unsafe_allow_html=True,
+    )
+
+    asking, filtering = st.tabs(["Ask a question", "Filter documents"])
+
+    with filtering:
+        application_col_filter, output_col_filter = st.columns([1, 1.5])
+        # make the buttons text smaller
+        with application_col_filter:
+            st.markdown(
+                """
+                <p class="description">
+                This filter function allows you to see all documents that match the selected filters. The documents can be accessed via a link. \n
+                """,
+                unsafe_allow_html=True,
+            )
+            if st.button("Filter documents"):
+                filters, status = build_filter(
+                    meta_data=df_transformed,
+                    authors_filter=selected_authors,
+                    draft_cats_filter=selected_draft_cats,
+                )
+                if status == "no filters selected":
+                    st.info("No filters selected. All documents will be shown.")
+                    df_filtered = df[
+                        ["author", "href", "draft_labs", "round"]
+                    ].sort_values(by="author")
+                    trigger_filter = True
+                if status == "no results found":
+                    st.info(
+                        "No documents found for the combination of filters you've chosen. All countries are represented at least once in the data. Remove the draft categories to see all documents for the countries selected or try other draft categories."
+                    )
+                if status == "success":
+                    df_filtered = df[df["retriever_id"].isin(filters["retriever_id"])][
+                        ["author", "href", "draft_labs", "round"]
+                    ].sort_values(by="author")
+                    trigger_filter = True
+
+    with asking:
+        application_col_ask, output_col_ask = st.columns([1, 1.5])
+        with application_col_ask:
+            st.markdown(
+                """
+                <p class="description"> Ask a question, noting that the database has been restricted by filters and that your question should pertain to the selected data. \n
+                """,
+                unsafe_allow_html=True,
+            )
+            if "prompt" not in st.session_state:
+                prompt = st.text_area("Enter a question")
+            if (
+                "prompt" in st.session_state
+                and st.session_state.prompt in example_prompts  # noqa: E501
+            ):  # noqa: E501
+                prompt = st.text_area(
+                    "Enter a question", value=st.session_state.prompt
+                )  # noqa: E501
+            if (
+                "prompt" in st.session_state
+                and st.session_state.prompt not in example_prompts  # noqa: E501
+            ):  # noqa: E501
+                del st.session_state["prompt"]
+                prompt = st.text_area("Enter a question")
+
+            trigger_ask = st.session_state.setdefault("trigger", False)
+
+            if st.button("Ask"):
+                if prompt == "":
+                    st.error(
+                        "Please enter a question. Reloading the app in few seconds"
+                    )
+                    time.sleep(3)
+                    st.rerun()
+                with st.spinner("Filtering data...") as status:
+                    filter_selection_transformed, status = build_filter(
+                        meta_data=df_transformed,
+                        authors_filter=selected_authors,
+                        draft_cats_filter=selected_draft_cats,
+                    )
+
+                    if status == "no filters selected":
+                        st.info(
+                            "No filters selcted.This will increase the prcessing time significantly. Please select at least one filter."
+                        )
+                        # st.error(
+                        #     "Selecting a filter is mandatory. We especially recommend to select countries you are interested in. Selecting at least one filter is mandatory, because otherwise the model would have to analyze all available documents which results in inaccurate answers and long processing times. Please select at least one filter."
+                        # )
+                        # st.stop()
+
+                    documents = pipeline.document_store.get_all_documents(
+                        filters=filter_selection_transformed
+                    )
+
+                    st.success("Filtering data completed.")
+                with st.spinner("Answering question...") as status:
+                    if filter_selection_transformed == {}:
+                        st.warning(
+                            "The combination of filters you've chosen does not match any documents. Giving answer based on all documents. Please note that the answer might not be accurate. We highly recommend to use a combination of filters that match the data. All countries are represented at least once in the data. Thus, for example, you could remove the draft categories to match the documents. Or you could check with the Filter documents function which documents are available for the selected countries by removing the draft categories and filter the documents."
+                        )
+
+                    result = pipeline.run(
+                        prompt=prompt, filters=filter_selection_transformed
+                    )
+                    trigger_ask = True
+                    st.success("Answering question completed.")
+
+            st.markdown("### Examples")
+            for i, prompt in enumerate(example_prompts):
+                # with col[i % 4]:
+                if st.button(prompt):
+                    if "key" not in st.session_state:
+                        st.session_state["prompt"] = prompt
+            st.markdown(
+                """
+                <ul class="description" style="font-size: 20px;">
+                    <li style="font-size: 17px;">These are example prompts that can be used to ask questions to the model</li>
+                    <li style="font-size: 17px;">Click on a prompt to use it as a question. You can also type your own question in the text area above.</li>
+                    <li style="font-size: 17px;">For questions like "How do country a, b and c [...]" please make sure to select the countries in the filter section. Otherwise the answer will not be accurate. In general we highly recommend to use the filter functions to narrow down the data.</li>
+                </ul>
+                """,
+                unsafe_allow_html=True,
+            )
+
+            # for i, prompt in enumerate(example_prompts):
+            #     # with col[i % 4]:
+            #     if st.button(prompt):
+            #         if "key" not in st.session_state:
+            #             st.session_state["prompt"] = prompt
+            # Define the button
+
+    if trigger_ask:
+        with output_col_ask:
+            if result is None:
+                st.error(
+                    "Open AI rate limit exceeded. Please try again in a few seconds."
+                )
+                st.stop()
+            meta_data = get_meta(result=result)
+            answer = result["answers"][0].answer
+
+            meta_data_cleaned = []
+            seen_retriever_ids = set()
+
+            for data in meta_data:
+                retriever_id = data["retriever_id"]
+                content = data["content"]
+                if retriever_id not in seen_retriever_ids:
+                    meta_data_cleaned.append(
+                        {
+                            "retriever_id": retriever_id,
+                            "href": data["href"],
+                            "content": [content],
+                        }
+                    )
+                    seen_retriever_ids.add(retriever_id)
+                else:
+                    for i, item in enumerate(meta_data_cleaned):
+                        if item["retriever_id"] == retriever_id:
+                            meta_data_cleaned[i]["content"].append(content)
+
+            references = ["\n"]
+            for data in meta_data_cleaned:
+                retriever_id = data["retriever_id"]
+                href = data["href"]
+                references.append(f"-[{retriever_id}]: {href} \n")
+            st.write("#### 📌 Answer")
+            typewriter(
+                text=answer,
+                references=references,
+                speed=100,
+            )
+
+            with st.expander("Show more information to the documents"):
+                for data in meta_data_cleaned:
+                    markdown_text = f"- Document: {data['retriever_id']}\n"
+                    markdown_text += "    - Text passages\n"
+                    for content in data["content"]:
+                        content = (
+                            content.replace("[", "").replace("]", "").replace("'", "")
+                        )
+                        content = " ".join(content.split())
+                        markdown_text += f"        - {content}\n"
+                    st.write(markdown_text)
+
+        trigger = 0
+
+    if trigger_filter:
+        with output_col_filter:
+            st.markdown("### Overview of all filtered documents")
+            st.dataframe(
+                df_filtered,
+                hide_index=True,
+                column_config={
+                    "author": st.column_config.ListColumn("Authors"),
+                    "href": st.column_config.LinkColumn("Link to Document"),
+                    "draft_labs": st.column_config.ListColumn("Draft Categories"),
+                    "round": st.column_config.NumberColumn("Round"),
+                },
+            )

src/data_processing/document_store_data.py

@@ -0,0 +1,93 @@
+import pandas as pd
+import ast
+import json
+
+DATASET = "data/inc_df_v6_small_4.csv"
+DATASET_PROCESSED = "data/inc_df.csv"
+MEMBERS = "data/authors_filter.json"
+
+
+def main():
+    print(f"Length of dataset: {len(pd.read_csv(DATASET))}")
+    df = pd.read_csv(DATASET)
+    df["retriever_id"] = df.index
+    columns = [
+        "retriever_id",
+        "description",
+        "href",
+        "draft_labs_list",
+        "authors_list",
+        "draft_allcats",
+        "doc_subtype",
+        "doc_type",
+        "text",
+        "round",
+    ]
+
+    df = df[columns]
+
+    df.rename(
+        mapper={
+            "draft_labs_list": "draft_labs",
+            "draft_allcats": "draft_cats",
+            "authors_list": "author",
+        },
+        axis=1,
+        inplace=True,
+    )
+
+    ###Subselect for countries and country groups
+    with open(MEMBERS, "r") as f:
+        authors = json.load(f)
+    special_character_words_mapper = {
+        "Côte D'Ivoire": "Cote DIvoire",
+        "Ligue Camerounaise Des Droits De L'Homme": "Ligue Camerounaise Des Droits De LHomme",
+        "Association Pour L'Integration Et La Developpement Durable Au Burundi": "Association Pour LIntegration Et La Developpement Durable Au Burundi",
+    }
+    members = [
+        authors[key]
+        for key in [
+            "Members - Countries",
+            "Members - International and Regional State Associations",
+        ]
+    ]
+    members = [item for sublist in members for item in sublist]
+    members = [special_character_words_mapper.get(member, member) for member in members]
+
+    nonmembers = [
+        authors[key]
+        for key in [
+            "Intergovernmental Negotiation Committee",
+            "Observers and Other Participants",
+        ]
+    ]
+    nonmembers = [item for sublist in nonmembers for item in sublist]
+
+    df["author"][df["author"] == "['Côte D'Ivoire']"] = "['Cote DIvoire']"
+    df["author"][
+        df["author"] == "['Ligue Camerounaise Des Droits De L'Homme']"
+    ] = "['Ligue Camerounaise Des Droits De LHomme']"
+    df["author"][
+        df["author"]
+        == "['Association Pour L'Integration Et La Developpement Durable Au Burundi']"
+    ] = "['Association Pour LIntegration Et La Developpement Durable Au Burundi']"
+
+    df["author"] = df["author"].apply(ast.literal_eval)
+    df = df[df["author"].apply(lambda x: any(item in members for item in x))]
+    df["author"] = df["author"].apply(
+        lambda x: [item for item in x if item not in nonmembers]
+    )
+    df["author"] = df["author"].apply(
+        lambda x: [item.replace("Côte DIvoire", "Cote D'Ivoire") for item in x]
+    )
+    df["draft_labs"] = df["draft_labs"].fillna("[]")
+    df["author"][
+        df["author"] == "['The Alliance Of Small Island States (AOSIS)']"
+    ] = "['Alliance Of Small Island States (AOSIS)']"
+
+    print(f"Filtered dataset to {len(df)} entries")
+    df.to_csv(DATASET_PROCESSED, index=False)
+
+
+if __name__ == "__main__":
+    main()

src/data_processing/document_store_data_all.py

@@ -0,0 +1,93 @@
+import pandas as pd
+import ast
+import json
+
+DATASET = "data/inc_df_v6_small.csv"
+DATASET_PROCESSED = "data/inc_df.csv"
+MEMBERS = "data/authors_filter.json"
+
+
+def main():
+    print(f"Length of dataset: {len(pd.read_csv(DATASET))}")
+    df = pd.read_csv(DATASET)
+    df["retriever_id"] = df.index
+    columns = [
+        "retriever_id",
+        "description",
+        "href",
+        "draft_labs_list",
+        "authors_list",
+        "draft_allcats",
+        "doc_subtype",
+        "doc_type",
+        "text",
+        "round",
+    ]
+
+    df = df[columns]
+
+    df.rename(
+        mapper={
+            "draft_labs_list": "draft_labs",
+            "draft_allcats": "draft_cats",
+            "authors_list": "author",
+        },
+        axis=1,
+        inplace=True,
+    )
+
+    ###Subselect for countries and country groups
+    with open(MEMBERS, "r") as f:
+        authors = json.load(f)
+    special_character_words_mapper = {
+        "Côte D'Ivoire": "Côte DIvoire",
+        "Ligue Camerounaise Des Droits De L'Homme": "Ligue Camerounaise Des Droits De LHomme",
+        "Association Pour L'Integration Et La Developpement Durable Au Burundi": "Association Pour LIntegration Et La Developpement Durable Au Burundi",
+    }
+    members = [
+        authors[key]
+        for key in [
+            "Members - Countries",
+            "Members - International and Regional State Associations",
+        ]
+    ]
+    members = [item for sublist in members for item in sublist]
+    members = [special_character_words_mapper.get(member, member) for member in members]
+
+    nonmembers = [
+        authors[key]
+        for key in [
+            "Intergovernmental Negotiation Committee",
+            "Observers and Other Participants",
+        ]
+    ]
+    nonmembers = [item for sublist in nonmembers for item in sublist]
+
+    df["author"][df["author"] == "['Côte D'Ivoire']"] = "['Côte DIvoire']"
+    df["author"][
+        df["author"] == "['Ligue Camerounaise Des Droits De L'Homme']"
+    ] = "['Ligue Camerounaise Des Droits De LHomme']"
+    df["author"][
+        df["author"]
+        == "['Association Pour L'Integration Et La Developpement Durable Au Burundi']"
+    ] = "['Association Pour LIntegration Et La Developpement Durable Au Burundi']"
+
+    df["author"] = df["author"].apply(ast.literal_eval)
+    df = df[df["author"].apply(lambda x: any(item in members for item in x))]
+    df["author"] = df["author"].apply(
+        lambda x: [item for item in x if item not in nonmembers]
+    )
+    df["author"] = df["author"].apply(
+        lambda x: [item.replace("Côte DIvoire", "Côte D 'Ivoire") for item in x]
+    )
+    df["draft_labs"] = df["draft_labs"].fillna("[]")
+    df["author"][
+        df["author"] == "['The Alliance Of Small Island States (AOSIS)']"
+    ] = "['Alliance Of Small Island States (AOSIS)']"
+
+    print(f"Filtered dataset to {len(df)} entries")
+    df.to_csv(DATASET_PROCESSED, index=False)
+
+
+if __name__ == "__main__":
+    main()

src/data_processing/get_meta_data_filter.py

@@ -0,0 +1,21 @@
+import pandas as pd
+import sys
+import os
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
+from src.rag.pipeline import RAGPipeline
+
+DATASET = os.path.join("data", "inc_df.csv")
+META_DATA = os.path.join("database", "meta_data.csv")
+
+rag_pipeline = RAGPipeline(
+    embedding_model="sentence-transformers/distiluse-base-multilingual-cased-v1",
+    prompt_template="src/rag/prompt_template.yaml",
+)
+
+meta_data = pd.DataFrame(
+    [document.meta for document in rag_pipeline.document_store.get_all_documents()]
+)
+
+meta_data = meta_data.drop_duplicates(subset=["retriever_id"], keep="first")
+
+meta_data.to_csv(META_DATA, index=False)

src/data_processing/taxonomy_processing.py

@@ -0,0 +1,47 @@
+import os
+
+from src.utils.data import load_json, save_json
+
+AUTHORS_TAXONOMY = os.path.join("data", "authors_taxonomy.json")
+AUTHORS_FILTER = os.path.join("data", "authors_filter.json")
+
+DRAFT_CATEGORIES_TAXONOMY = os.path.join("data", "draftcat_taxonomy.json")
+DRAFT_CATEGORIES_FILTER = os.path.join("data", "draftcat_taxonomy_filter.json")
+
+
+def get_authors(taxonomy: dict) -> dict:
+    countries = taxonomy["Members"]["Countries"]
+    associations = taxonomy["Members"][
+        "International and Regional State Associations"
+    ]  # noqa: E501
+    intergovernmental_negotiations = taxonomy[
+        "Intergovernmental Negotiation Committee"
+    ]  # noqa: E501
+    observers = taxonomy["Observers and Other Participants"]  # noqa: E501
+    return {
+        "Members - Countries": countries,
+        "Members - International and Regional State Associations": associations,  # noqa: E501
+        "Intergovernmental Negotiation Committee": intergovernmental_negotiations,  # noqa: E501
+        "Observers and Other Participants": observers,
+    }
+
+
+def get_draftcategories(taxonomy: dict) -> dict:
+    taxonomy_filter = {}
+    for draft_part, part_values in taxonomy.items():
+        part = draft_part
+        temp_values = []
+        for part_name, part_value in part_values.items():
+            temp_values.append(part_value)
+        taxonomy_filter[part] = temp_values
+    return taxonomy_filter
+
+
+if __name__ == "__main__":
+    authors_taxonomy = load_json(AUTHORS_TAXONOMY)
+    authors_filter = get_authors(authors_taxonomy)
+    save_json(file_path=AUTHORS_FILTER, data=authors_filter)
+
+    draft_categories_taxonomy = load_json(DRAFT_CATEGORIES_TAXONOMY)
+    draft_categories_filter = get_draftcategories(draft_categories_taxonomy)
+    save_json(file_path=DRAFT_CATEGORIES_FILTER, data=draft_categories_filter)

src/document_store/document_store.py

@@ -0,0 +1,180 @@
+from haystack.document_stores import InMemoryDocumentStore
+import pandas as pd
+import os
+import pathlib
+import ast
+from sklearn.preprocessing import MultiLabelBinarizer
+from langchain_community.document_loaders import DataFrameLoader
+from langchain.text_splitter import (
+    RecursiveCharacterTextSplitter,
+)
+from typing import Any
+
+
+INC_TEST_DATASET_PATH = os.path.join("data", "inc_df.csv")
+EMBEDDING_DIMENSION = 512
+
+special_character_words_mapper = {
+    "Côte D'Ivoire": "Côte DIvoire",
+    "Ligue Camerounaise Des Droits De L'Homme": "Ligue Camerounaise Des Droits De LHomme",
+    "Association Pour L'Integration Et La Developpement Durable Au Burundi": "Association Pour LIntegration Et La Developpement Durable Au Burundi",
+}
+special_character_words_reverse_mapper = {}
+for key, value in special_character_words_mapper.items():
+    special_character_words_reverse_mapper[value] = key
+
+
+def transform_to_list(row):
+    special_characters = False
+    if str(row) == "[]" or str(row) == "nan":
+        return []
+    else:
+        # replace special characters
+        for key, value in special_character_words_mapper.items():
+            if key in row:
+                row = row.replace(key, value)
+                special_characters = True
+        row = ast.literal_eval(row)
+        if special_characters:
+            for key, value in special_character_words_reverse_mapper.items():
+                if key in row:
+                    # get the index of the special character word
+                    index = row.index(key)
+                    # replace the special character word with the original word
+                    row[index] = value
+        return row
+
+
+def transform_data(df: pd.DataFrame):
+    # df["author"] = df["authors"].drop(columns=["authors"], axis=1)
+    df = df[df["doc_subtype"] != "Working documents"]
+    df = df[df["doc_subtype"] != "Contact Groups"]
+    df = df[df["doc_subtype"] != "Unsolicitated Submissions"]
+    df = df[df["doc_type"] != "official document"]
+    df = df[df["doc_subtype"] != "Stakeholder Dialogue"]
+    df["text"] = df["text"].astype(str).str.replace("_x000D_", " ")
+    df["text"] = df["text"].astype(str).str.replace("\n", " ")
+    # df["text"] = df["text"].astype(str).str.replace("\r", " ")
+    df["author"] = df["author"].str.replace("\xa0", " ")
+    df["author"] = df["author"].str.replace("ü", "u")
+    df["author"] = df["author"].str.strip()
+    df["author"] = df["author"].astype(str).str.replace("\r", " ")
+
+    df = df[
+        [
+            "author",
+            "doc_type",
+            "round",
+            "text",
+            "href",
+            "draft_labs",
+            "draft_cats",
+            "retriever_id",
+        ]
+    ].copy()
+
+    df = df.rename(columns={"text": "page_content"}).copy()
+
+    df["draft_labs2"] = df["draft_labs"]
+    df["author2"] = df["author"]
+
+    df["draft_labs"] = df.apply(lambda x: transform_to_list(x["draft_labs"]), axis=1)
+    df["author"] = df.apply(lambda x: transform_to_list(x["author"]), axis=1)
+
+    # df["draft_labs"] = df["draft_labs"].apply(
+    #     lambda x: ast.literal_eval(x) if str(x) != "[]" or str(x) != "nan" else []
+    # )
+    # df["author"] = df["author"].apply(
+    #     lambda x: ast.literal_eval(x) if str(x) != "[]" else []
+    # )
+
+    mlb = MultiLabelBinarizer(sparse_output=True)
+    mlb = MultiLabelBinarizer()
+    df = df.join(
+        pd.DataFrame(
+            mlb.fit_transform(df.pop("draft_labs")),
+            columns=mlb.classes_,
+            index=df.index,
+        )
+    ).join(
+        pd.DataFrame(
+            mlb.fit_transform(df.pop("author")), columns=mlb.classes_, index=df.index
+        )
+    )
+
+    df["draft_labs"] = df["draft_labs2"]
+    df = df.drop(columns=["draft_labs2"], axis=1)
+
+    df["author"] = df["author2"]
+    df = df.drop(columns=["author2"], axis=1)
+
+    loader = DataFrameLoader(df, page_content_column="page_content")
+    docs = loader.load()
+    return docs
+
+
+def process_data(docs):
+
+    chunk_size = 512
+    text_splitter = RecursiveCharacterTextSplitter(
+        chunk_size=chunk_size,
+        chunk_overlap=int(chunk_size / 10),
+        add_start_index=True,
+        strip_whitespace=True,
+        separators=["\n\n", "\n", " ", ""],
+    )
+
+    docs_chunked = text_splitter.transform_documents(docs)
+
+    df = pd.DataFrame(docs_chunked, columns=["page_content", "metadata", "type"]).drop(
+        "type", axis=1
+    )
+    df["page_content"] = df["page_content"].astype(str)
+    df["page_content"] = df["page_content"].str.replace("'page_content'", "")
+    df["page_content"] = df["page_content"].str.replace("(", "")
+    df["page_content"] = df["page_content"].str.replace(")", "").str[1:]
+    df = pd.concat(
+        [df.drop("metadata", axis=1), df["metadata"].apply(pd.Series)], axis=1
+    )
+    df = df.rename(columns={0: "a", 1: "b"})
+    df = pd.concat([df.drop(["a", "b"], axis=1), df["b"].apply(pd.Series)], axis=1)
+
+    cols = ["author", "draft_labs"]
+    for c in cols:
+        df[c] = df[c].apply(
+            lambda x: "".join(x) if isinstance(x, (list, tuple)) else str(x)
+        )
+        chars = ["[", "]", "'"]
+        for g in chars:
+            df[c] = df[c].str.replace(g, "")
+
+    df["page_content"] = df["page_content"].astype(str).str.replace("\n", " ")
+    df["page_content"] = df["page_content"].astype(str).str.replace("\r", " ")
+
+    cols = ["author", "draft_labs", "page_content"]
+    df["page_content"] = df[cols].apply(lambda row: " | ".join(row.astype(str)), axis=1)
+    df = df.rename(columns={"page_content": "content"})
+
+    documents = []
+    for _, row in df.iterrows():
+        row_meta: dict[str, Any] = {}
+        for column in df.columns:
+            if column != "content":
+                if column == "retriever_id":
+                    row_meta[column] = str(row[column])
+                else:
+                    row_meta[column] = row[column]
+        documents.append({"content": row["content"], "meta": row_meta})
+    return documents
+
+
+def get_document_store():
+    df = pd.read_csv(INC_TEST_DATASET_PATH)
+    # df["retriever_id"] = [str(i) for i in range(len(df))]
+    pathlib.Path("database").mkdir(parents=True, exist_ok=True)
+    document_store = InMemoryDocumentStore(
+        embedding_field="embedding", embedding_dim=EMBEDDING_DIMENSION, use_bm25=False
+    )
+    docs = transform_data(df=df)
+    document_store.write_documents(process_data(docs=docs))
+    return document_store

src/rag/pipeline.py

@@ -0,0 +1,93 @@
+import os
+import pickle
+from typing import Any
+
+from dotenv import load_dotenv
+from haystack.nodes import (  # type: ignore
+    AnswerParser,
+    EmbeddingRetriever,
+    PromptNode,
+    PromptTemplate,
+)
+from haystack.pipelines import Pipeline
+
+from src.document_store.document_store import get_document_store
+
+load_dotenv()
+
+OPENAI_API_KEY = os.environ.get("OPEN_API_KEY")
+
+
+class RAGPipeline:
+    def __init__(
+        self,
+        embedding_model: str,
+        prompt_template: str,
+    ):
+        self.load_document_store()
+        self.embedding_model = embedding_model
+        self.prompt_template = prompt_template
+        self.retriever_node = self.generate_retriever_node()
+        self.prompt_node = self.generate_prompt_node()
+        self.update_embeddings()
+        self.pipe = self.build_pipeline()
+
+    def run(self, prompt: str, filters: dict) -> Any:
+        try:
+            result = self.pipe.run(query=prompt, params={"filters": filters})
+            return result
+        except Exception as e:
+            print(e)
+            return None
+
+    def build_pipeline(self):
+        pipe = Pipeline()
+        pipe.add_node(component=self.retriever_node, name="retriever", inputs=["Query"])
+        pipe.add_node(
+            component=self.prompt_node,
+            name="prompt_node",
+            inputs=["retriever"],
+        )
+        return pipe
+
+    def load_document_store(self):
+        if os.path.exists(os.path.join("database", "document_store.pkl")):
+            with open(
+                file=os.path.join("database", "document_store.pkl"), mode="rb"
+            ) as f:
+                self.document_store = pickle.load(f)
+        else:
+            self.document_store = get_document_store()
+
+    def generate_retriever_node(self):
+        retriever_node = EmbeddingRetriever(
+            document_store=self.document_store,
+            embedding_model=self.embedding_model,
+            top_k=7,
+        )
+        return retriever_node
+
+    def update_embeddings(self):
+        if not os.path.exists(os.path.join("database", "document_store.pkl")):
+            self.document_store.update_embeddings(
+                self.retriever_node, update_existing_embeddings=True
+            )
+
+            with open(
+                file=os.path.join("database", "document_store.pkl"), mode="wb"
+            ) as f:
+                pickle.dump(self.document_store, f)
+
+    def generate_prompt_node(self):
+        rag_prompt = PromptTemplate(
+            prompt=self.prompt_template,
+            output_parser=AnswerParser(reference_pattern=r"Document\[(\d+)\]"),
+        )
+        prompt_node = PromptNode(
+            model_name_or_path="gpt-4",
+            default_prompt_template=rag_prompt,
+            api_key="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",
+            max_length=3000,
+            model_kwargs={"temperature": 0.2, "max_tokens": 4096},
+        )
+        return prompt_node

src/rag/prompt_template.yaml

@@ -0,0 +1,20 @@
+name: deepset/question-answering-with-document-references
+text: |
+  Answer the question '{query}' using only the provided documents and avoiding text. 
+  Formulate your answer in the style of an academic report. 
+  Provide example quotes and citations using extracted text from the documents.
+  Use facts and numbers from the documents in your answer. 
+  ALWAYS include the references of the documents used from documents at the end of each applicable sentence using the format [number].
+  If the answer isn't in the document say 'Answering is not possible given the available information'.
+  Documents: \n
+  {join(documents, delimiter=new_line, pattern=new_line+'Document($retriever_id): $content', str_replace={new_line: ' ', '[': '(', ']': ')'})} \n
+  Answer: 
+tags:
+  - question-answering
+description: Perform question answering with references to documents.
+meta:
+  authors:
+    - deepset-ai
+version: '0.1.0'
+
+

src/utils/data.py

@@ -0,0 +1,76 @@
+import json
+from typing import Any
+
+import pandas as pd
+import streamlit as st
+from functools import reduce
+
+
+def get_filter_values(df: pd.DataFrame, column_name: str) -> list:
+    return df[column_name].unique().tolist()
+
+
+def build_filter(
+    meta_data: pd.DataFrame,
+    authors_filter: list[str],
+    draft_cats_filter: list[str],
+    round_filter: list[int],
+) -> dict[str, int | str] | dict:
+    authors = authors_filter
+    round_number = round_filter
+    draft_cats = draft_cats_filter
+
+    # set authors_flag to True if not empty list
+    authors_flag = True if len(authors) > 0 else False
+    draft_cats_flag = True if len(draft_cats) > 0 else False
+    round_number_flag = True if len(round_number) > 0 else False
+
+    conditions = []
+
+    if authors_flag:
+        authors_condition = (meta_data[col] == 1 for col in authors)
+        authors_conditions_list = reduce(lambda a, b: a | b, authors_condition)
+        conditions.append(authors_conditions_list)
+
+    if draft_cats_flag:
+        draft_cat_condition = (meta_data[col] for col in draft_cats)
+        draft_cat_conditions_list = reduce(lambda a, b: a | b, draft_cat_condition)
+        conditions.append(draft_cat_conditions_list)
+
+    if round_number_flag:
+        round_condition = meta_data["round"].isin(round_number)
+        conditions.append(round_condition)
+
+    if len(conditions) == 0:
+        filtered_retriever_ids = []
+    else:
+        final_condition = reduce(lambda a, b: a & b, conditions)
+        filtered_retriever_ids = meta_data[final_condition]["retriever_id"].tolist()
+    if len(filtered_retriever_ids) == 0:
+        return {}
+    else:
+        return {"retriever_id": filtered_retriever_ids}
+
+
+def load_json(file_path: str) -> dict:
+    with open(file_path, "r") as f:
+        return json.load(f)
+
+
+def save_json(file_path: str, data: dict) -> None:
+    with open(file_path, "w") as f:
+        json.dump(data, f, indent=4)
+
+
+def get_meta(result: dict[str, Any]) -> list[dict[str, Any]]:
+    meta_data = []
+    for doc in result["documents"]:
+        current_meta = doc.meta
+        current_meta["content"] = doc.content
+        meta_data.append(current_meta)
+    return meta_data
+
+
+def load_css(file_name) -> None:
+    with open(file_name) as f:
+        st.markdown(f"<style>{f.read()}</style>", unsafe_allow_html=True)

src/utils/data_v2.py

@@ -0,0 +1,80 @@
+import json
+from typing import Any
+
+import pandas as pd
+import streamlit as st
+from functools import reduce
+
+
+def get_filter_values(df: pd.DataFrame, column_name: str) -> list:
+    return df[column_name].unique().tolist()
+
+
+def build_filter(
+    meta_data: pd.DataFrame,
+    authors_filter: list[str],
+    draft_cats_filter: list[str],
+ #   round_filter: list[int],
+) -> dict[str, int | str] | dict:
+    authors = authors_filter
+    #round_number = round_filter
+    draft_cats = draft_cats_filter
+
+    # set authors_flag to True if not empty list
+    authors_flag = True if len(authors) > 0 else False
+    draft_cats_flag = True if len(draft_cats) > 0 else False
+    #round_number_flag = True if len(round_number) > 0 else False
+
+    if authors_flag is False and draft_cats_flag is False:
+        return {}, "no filters selected"
+
+    conditions = []
+
+    if authors_flag:
+        authors_condition = (meta_data[col] == 1 for col in authors)
+        authors_conditions_list = reduce(lambda a, b: a | b, authors_condition)
+        conditions.append(authors_conditions_list)
+
+    if draft_cats_flag:
+        draft_cat_condition = (meta_data[col] for col in draft_cats)
+        draft_cat_conditions_list = reduce(lambda a, b: a | b, draft_cat_condition)
+        conditions.append(draft_cat_conditions_list)
+
+    # if round_number_flag:
+    #     round_condition = meta_data["round"].isin(round_number)
+    #     conditions.append(round_condition)
+
+    if len(conditions) == 0:
+        filtered_retriever_ids = []
+    else:
+        final_condition = reduce(lambda a, b: a & b, conditions)
+        filtered_retriever_ids = meta_data[final_condition]["retriever_id"].tolist()
+
+    if len(filtered_retriever_ids) == 0:
+        return {}, "no results found"
+    else:
+        return {"retriever_id": filtered_retriever_ids}, "success"
+
+
+def load_json(file_path: str) -> dict:
+    with open(file_path, "r") as f:
+        return json.load(f)
+
+
+def save_json(file_path: str, data: dict) -> None:
+    with open(file_path, "w") as f:
+        json.dump(data, f, indent=4)
+
+
+def get_meta(result: dict[str, Any]) -> list[dict[str, Any]]:
+    meta_data = []
+    for doc in result["documents"]:
+        current_meta = doc.meta
+        current_meta["content"] = doc.content
+        meta_data.append(current_meta)
+    return meta_data
+
+
+def load_css(file_name) -> None:
+    with open(file_name) as f:
+        st.markdown(f"<style>{f.read()}</style>", unsafe_allow_html=True)

src/utils/writer.py

@@ -0,0 +1,17 @@
+import streamlit as st
+import time
+
+
+def typewriter(text: str, references: list, speed: int):
+    tokens = text.split()
+    container = st.empty()
+    for index in range(len(tokens) + 1):
+        curr_full_text = " ".join(tokens[:index])
+        container.markdown(curr_full_text)
+        time.sleep(1 / speed)
+    curr_full_text += "\n"
+    container.markdown(curr_full_text)
+    curr_full_text += "\n **References** \n"
+    container.markdown(curr_full_text)
+    curr_full_text += "\n".join(references)
+    container.markdown(curr_full_text)

style/style.css

@@ -0,0 +1,103 @@
+.remark {
+    background-color: #FFCDD2;
+    border: 1px solid #E57373;
+    border-radius: 4px;
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+    margin-bottom: 1rem;
+    padding: 1rem;
+}
+
+.remark-content {
+    display: flex;
+    align-items: center;
+}
+
+.remark-text {
+    margin-bottom: 0.5rem;
+    font-size: 20;
+}
+
+.feedback-link {
+    display: inline-block;
+    background-color: #b50d1c; /* Adjusted color */
+    color: white;
+    border: none;
+    border-radius: 4px;
+    padding: 0.5rem 1rem;
+    text-decoration: none;
+    font-size: 1rem;
+    cursor: pointer;
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); /* Added */
+}
+
+.feedback-link:hover {
+    background-color: #8c0716; /* Adjusted hover color */
+}
+
+.feedback-link[href^="mailto"] {
+    color: white !important;
+    text-decoration: none !important;
+}
+
+
+
+/* Style streamlit general text */
+.description {
+    font-size:20px !important;
+}
+
+
+/* Style streamlit header with bold text  */
+.header {
+    font-size:30px !important;
+    font-weight: bold;
+}
+
+.stMultiSelect > div > div > div {
+    width: 350px !important;
+    font-size: 20px;
+}
+
+/* Style columns */
+[data-testid="column"] {
+    border-radius: 15px;
+       background-color: white;
+       box-shadow: 0 0 10px #eee;
+       border: 1px solid #ddd;
+       padding: 1rem;;
+}
+/* Style containers */
+[data-testid="stVerticalBlock"] > [style*="flex-direction: column;"] > [data-testid="stVerticalBlock"] {
+    border-radius: 15px;
+       background-color: white;
+       box-shadow: 0 0 10px #eee;
+       border: 1px solid #ddd;
+       padding: 1rem;;
+}
+
+.stTabs [data-baseweb="tab-list"] button [data-testid="stMarkdownContainer"] p {
+    font-size:20px;
+    }
+
+/* Style streamlit button */
+.stButton>button {
+    font-size: 12px;
+    padding: 8px 12px;
+    border: none;
+    text-align: center;
+    text-decoration: none;
+    display: inline-block;
+    cursor: pointer;
+    border-radius: 4px;
+    transition: background-color 0.3s, box-shadow 0.3s; 
+    background-color: #f2f2f2;
+    color: #333;
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); 
+
+.stButton>button:hover {
+    background-color: #e0e0e0;
+    box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2); 
+}
+
+
+