<jupyter_start><jupyter_text>Aviationstack  Objective:* Understand what kind of data is dumping into the dataset* showcase how to connect to postgres db using pandas  Add Helpers Libs to Python Path<jupyter_code>%%sh
cd /home/analytics/workspace/
pip3 install -e .<jupyter_output>Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/analytics/workspace
Requirement already satisfied: psycopg2-binary==2.8.6 in /home/analytics/.local/lib/python3.8/site-packages (from avstackhelper==0.0.1) (2.8.6)
Installing collected packages: avstackhelper
  Attempting uninstall: avstackhelper
    Found existing installation: avstackhelper 0.0.1
    Uninstalling avstackhelper-0.0.1:
      Successfully uninstalled avstackhelper-0.0.1
  Running setup.py develop for avstackhelper
Successfully installed avstackhelper<jupyter_text>Restart Kernel  Libs Versioning<jupyter_code>from IPython import __version__ as ipython_version
from pandas import __version__ as pandas_version
from psycopg2 import __version__ as psycopg2_version
from avstackhelper import __version__ as avstackhelper_version

VERSION = " Versioning "
IPYTHON = " IPython: %s " % ipython_version
PSYCOPG2 = " psycopg2: %s " % psycopg2_version
PANDAS = " pandas: %s " % pandas_version
ASHELPER = "AVSTACKHELPER %s" % avstackhelper_version

print(VERSION.center(60, '#'))
print(IPYTHON.center(60, "-"))
print(PSYCOPG2.center(60, "-"))
print(PANDAS.center(60, "-"))
print(ASHELPER.center(60, "-"))
print("".center(60, '#'))<jupyter_output>######################## Versioning ########################
--------------------- IPython: 7.19.0 ----------------------
---------- psycopg2: 2.8.6 (dt dec pq3 ext lo64) -----------
---------------------- pandas: 1.2.1 -----------------------
--------------------AVSTACKHELPER 0.0.1---------------------
############################################################<jupyter_text>Import Libs<jupyter_code>import pandas as pd
from avstackhelper.etl.session import postgresql_connect
from avstackhelper.etl.session import postgresql_fetch_data<jupyter_output><empty_output><jupyter_text>Intiatiate DB Connection  Defining Globals Variables- DB Param<jupyter_code>param_dic = {
    "host"      : "aviationdb_postgres",
    "database"  : "aviationstack",
    "user"      : "testfligoo",
    "password"  : "<PASSWORD>"
}
TABLE_TO_FETCH = "t_active_flights_view"
conn = postgresql_connect(param_dic)<jupyter_output>Connecting to the PostgreSQL database...
Connection successful<jupyter_text>Data Retrieval<jupyter_code>column_names = [
    "flight_date", "flight_status", "airline_name", "flight_number",
    "departure_airport", "departure_timezone", "arrival_airport",
    "arrival_timezone", "arrival_terminal", "insert_timestamp"
]
SQL_QUERY = "SELECT {} FROM {}".format(",".join(column_names), TABLE_TO_FETCH)
SQL_QUERY
tupples = postgresql_fetch_data(conn, SQL_QUERY)
len(tupples)<jupyter_output><empty_output><jupyter_text>Convert to Pandas DataFrame<jupyter_code>df = pd.DataFrame(tupples, columns=column_names)
df.info()
df.head()<jupyter_output><empty_output><jupyter_text>Data Analysys<jupyter_code>df.flight_status.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.179375, 'fraction_numerical': 0.0178125, 'mean_word_length': 4.938775510204081, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '129711', 'license': 'MIT', 'n_tokens_mistral': 1101, 'n_tokens_neox': 1056, 'n_words': 210, 'path': 'notebooks/AviationStack - Flights.ipynb', 'repo_name': 'nitsugahcram/real-time-flight-status', 'size': '3198'}