<jupyter_start><jupyter_text>Read EGN Abfallkalender programmatically with PythonThis code enables you to read the EGN Abfallkalender (https://www.egn-abfallkalender.de/kalender) programmatically with Python. The EGN Abfallkalender offers appointments for the following German cities:* Brüggen* Dormagen* Gefrath* Grevenbroich* Nettetal* Niederkrüchten* Schwalmtal* Tönisvorst* WillichYou can use it to integrate the EGN appointments e.g. within your own Smart Home solution.Note: * This is only a small excerpt from my entire solution for the EGN calendar integration. I focus only on the general mechanism to request the data from the calendar for subsequent use* My experience is also that the online calendar is sometimes not in sync with the paper-based calendar, in most cases the paper-based calendar "wins". I assume that this is only the result of a wrong (maybe also handmade?!) maintenance of the digital version... Installation1. Download this Jupyter notebook or copy the code e.g. within a new [Google Colab notebook](https://colab.research.google.com/)2. Replace the given address parts in line 19 - 22 with your personal address3. Run the code in your environment, and enjoy the output!<jupyter_code>"""
    Author: <NAME>
    License: MIT
    Email: <EMAIL>
"""

# External Imports
import requests
from bs4 import BeautifulSoup
import datetime
# Internal Imports
#NA

class EgnCalendar:
  def __init__(self):
    # Target URL
    self.EGN_URL = "https://www.egn-abfallkalender.de/kalender"

    # Replace the following lines with your specific data
    self.EGN_CITY = "Brüggen"
    self.EGN_DISTRICT = "Brüggen"
    self.EGN_STREET = "Hochstrasse"
    self.EGN_STREET_NUMBER = "68"

    # Determine current datetime and extract relevant datetime information
    now = datetime.datetime.now()
    self.current_year = now.year
    self.current_month = now.month
    self.current_day = now.day
    self.current_cweek = datetime.date(int(self.current_year), int(self.current_month), int(self.current_day)).isocalendar()[1]

    # Dictionary for en_de weekday mapping
    self.trans_dayname_en_de = {
      "Monday": "Montag",
      "Tuesday": "Dienstag",
      "Wednesday": "Mittwoch",
      "Thursday": "Donnerstag",
      "Friday": "Freitag",
      "Saturday": "Samstag",
      "Sunday": "Sonntag"
    }

    # Dictionary for EGN abbreviation color mapping
    self.trans_egnabbreviation_color = {
      "PPK": "Blaue Tonne",  # Blaue-Tonne
      "HML": "Graue Tonne",  # Graue-Tonne
      "LVP": "Gelbe Tonne",  # Gelbe-Tonne
      "BIO": "Bio Tonne",  # Bio-Tonne
      "WBM": "Bio Tonne"  # Bio-Tonne
    }

  # Determines and returns dayname from date information
  def date_to_dayname(self, input_date):
    day_name = datetime.datetime.strptime(input_date, '%Y-%m-%d')
    return day_name.strftime("%A")

  # Request the https://www.egn-abfallkalender.de/kalender page
  def request_calendar(self):
    try:
      ######  Perform first request ######
      # in order to retrieve xsrf-token, egn-session and csrf-token for second request

      # Step 1: Define request headers
      request_headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
        'Accept': '*/*'
      }

      # Step 2: Perform request
      response = requests.get(self.EGN_URL,
                              headers=request_headers,
                              verify=True)

      # Step 3: Process response
      # Get xsrf-token and egn-session from first request (coming from response)
      xsrf_token = response.cookies.get("XSRF-TOKEN")
      egn_abfallkalender_session = response.cookies.get("egn_abfallkalender_session")
      # Get csrf-token (coming from meta-tag in reponse body)
      soup = BeautifulSoup(response.text, features="html.parser")
      metatag_csrf_token = soup.find_all('meta', attrs={'name': 'csrf-token'})
      csrf_token = metatag_csrf_token[0].get("content")

      ######  Perform second request ######
      # in order to retrieve calendar data (in json format) we're interested in

      # Step 1: Define request cookies, headers and data
      request_cookies = {
        'XSRF-TOKEN': xsrf_token,
        'egn_abfallkalender_session': egn_abfallkalender_session
      }

      request_headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
        'Accept': '*/*',
        'X-CSRF-TOKEN': csrf_token,
      }

      request_data = {
        'city': self.EGN_CITY,
        'district': self.EGN_DISTRICT,
        'street': self.EGN_STREET,
        'street_number': self.EGN_STREET_NUMBER
      }

      # Step 2: Perform second request to retrieve calendar data
      response = requests.post(self.EGN_URL,
                               headers=request_headers,
                               cookies=request_cookies,
                               data=request_data)

      # Step 3: Process reponse
      # Convert reponse to json format
      # This is the original output coming from egn calendar request
      waste_discharge = response.json()

      # Create dictionary that holds our filtered calendar
      egn_appointments = dict()

      # Iterate through each element
      for month_iter in range(1, 13):
        current_item = waste_discharge["waste_discharge"][str(self.current_year)][str(month_iter)]
        # Consider only current month
        if self.current_month == month_iter:
          # Handle current month items
          for key, value in current_item.items():
            # Determine calender week of current item
            item_cweek = datetime.date(int(self.current_year), int(self.current_month), int(key)).isocalendar()[1]
            # Consider only current and next week calendar week items
            if (self.current_cweek == item_cweek) or (self.current_cweek+1 == item_cweek):
              date_key = str(self.current_year) + "-" + str(self.current_month).rjust(2, '0') + "-" + str(key).rjust(2, '0')
              egn_appointments.update({date_key: value})

      # ----------------> # Return (OK)
      return waste_discharge, egn_appointments

    except Exception as e:
      # ----------------> # Return (NOK)
      return f'{e}'


# Create EGN Calendar instance and request the calendar
egn_calendar = EgnCalendar()
egn_calendar_original, egn_calendar_filtered = egn_calendar.request_calendar()

# Print original output coming from egn calendar request
print("Original EGN Calendar Output:\n", egn_calendar_original)

# Print filtered output considers only current and next week calender items
print("\n\n")
print("Filtered output, this and next week calendar items:\n", egn_calendar_filtered)

# Print formatted text based on filtered calendar
print("\n\n")
print("Müllabfuhrtermine diese und nächste Woche:")
for key, value in egn_calendar_filtered.items():
  key_date = datetime.datetime.strptime(key, '%Y-%m-%d')
  current_weekday_name = egn_calendar.date_to_dayname(key)
  for i in value:
    # Skip hml2-value
    if i == "hml2":
      continue
    print(f"---> Am {egn_calendar.trans_dayname_en_de[current_weekday_name]} ({key}) wird die {egn_calendar.trans_egnabbreviation_color[str(value[0])]} abgeholt!")<jupyter_output>Original EGN Calendar Output:
 {'waste_discharge': {'2020': {'1': {'2': ['BIO'], '9': ['LVP'], '14': ['HML'], '15': ['BIO'], '16': ['PPK'], '23': ['LVP'], '28': ['HML'], '29': ['BIO']}, '2': {'6': ['LVP'], '11': ['HML'], '12': ['BIO'], '13': ['PPK'], '20': ['LVP'], '26': ['HML'], '27': ['BIO']}, '3': {'5': ['LVP'], '10': ['HML'], '11': ['BIO'], '12': ['PPK'], '19': ['LVP'], '24': ['HML'], '25': ['BIO']}, '4': {'2': ['LVP'], '6': ['HML'], '7': ['BIO'], '9': ['PPK'], '17': ['LVP'], '21': ['HML'], '22': ['BIO'], '29': ['LVP']}, '5': {'5': ['HML'], '6': ['BIO'], '7': ['PPK'], '14': ['LVP'], '19': ['HML'], '20': ['BIO'], '28': ['LVP']}, '6': {'3': ['HML', 'BIO'], '5': ['PPK'], '12': ['LVP'], '16': ['HML'], '17': ['BIO'], '25': ['LVP'], '30': ['HML']}, '7': {'1': ['BIO'], '2': ['PPK'], '9': ['LVP'], '14': ['HML'], '15': ['BIO'], '23': ['LVP'], '28': ['HML'], '29': ['BIO'], '30': ['PPK']}, '8': {'6': ['LVP'], '11': ['HML'], '12': ['BIO'], '20': ['LVP'], '25': ['HML'], '26': ['BIO'], '27': ['P[...]
{'fraction_non_alphanumeric': 0.15732654051431344, 'fraction_numerical': 0.02316836487142164, 'mean_word_length': 3.3601269169751453, 'pattern_counts': {'":': 13, '<': 6, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '67983', 'license': 'MIT', 'n_tokens_mistral': 2820, 'n_tokens_neox': 2663, 'n_words': 762, 'path': 'Read_EGN_Abfallkalender_Programmatically_With_Python.ipynb', 'repo_name': 'anjakuchenbecker/smarthome-egn-calendar', 'size': '8267'}