<jupyter_start><jupyter_text>Loading and Importing Data This section will import the data and display the first few rows of information.<jupyter_code>import numpy as np
import pandas as pd
import datetime as datetime
CDR_Descp = 'CDR_description.xlsx'
CDR = pd.read_excel(CDR_Descp)
CDR.head()
Cells_geo = pd.read_csv('cells_geo.csv', header = 0, index_col=0, delimiter=';')
Cells_geo.head()
Geo_desc = pd.read_excel('cells_geo_description.xlsx')
Geo_desc.head()
Telcom_ds1 = pd.read_csv('Telcom_dataset.csv')
Telcom_ds1.head()
Telcom_ds2 = pd.read_csv('Telcom_dataset2.csv')
Telcom_ds2.head()
Telcom_ds3 = pd.read_csv('Telcom_dataset3.csv')
Telcom_ds3.head()<jupyter_output><empty_output><jupyter_text>Data Interpretation This section will explore which type of information each data set contains for better manipulation<jupyter_code>CDR.columns
Cells_geo.columns
Geo_desc.columns
Telcom_ds1.columns
Telcom_ds2.columns
Telcom_ds3.columns
CDR.info()
Cells_geo.info()
Geo_desc.info()

Telcom_ds1.info()
Telcom_ds2.info()
Telcom_ds3.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5001 entries, 0 to 5000
Data columns (total 10 columns):
 #   Column           Non-Null Count  Dtype 
---  ------           --------------  ----- 
 0   PRODUCT          5001 non-null   object
 1   VALUE            5001 non-null   int64 
 2   DATE_TIME        5001 non-null   object
 3   CELL_ON_SITE     5001 non-null   object
 4   DW_A_NUMBER_INT  5001 non-null   object
 5   DW_B_NUMBER_INT  5001 non-null   object
 6   COUNTRY_A        5001 non-null   object
 7   COUNTRY_B        5001 non-null   object
 8   CELLID           5001 non-null   object
 9   SIET_ID          4396 non-null   object
dtypes: int64(1), object(9)
memory usage: 390.8+ KB<jupyter_text>Data Manipulation and Cleaning This section will find merge the data on common keys before cleaning for analysis<jupyter_code>Cells_geo.columns = map(str.lower, Cells_geo.columns)
Cells_geo.columns
Telcom_ds1.columns = map(str.lower, Telcom_ds1.columns)
columns = ['product', 'value', 'date_time', 'cell_on_site', 'dw_a_number_int','dw_b_number_int', 
           'country_a', 'country_b', 'cell_id', 'site_id']
Telcom_ds1.columns = columns
Telcom_ds1.columns
Telcom_ds2.columns = map(str.lower, Telcom_ds2.columns)
columns = ['product', 'value', 'date_time', 'cell_on_site', 'dw_a_number','dw_b_number', 
           'country_a', 'country_b', 'cell_id', 'site_id']
Telcom_ds2.columns = columns
Telcom_ds2.columns
Telcom_ds3.columns = map(str.lower, Telcom_ds3.columns)
columns = ['product', 'value', 'date_time', 'cell_on_site', 'dw_a_number_int','dw_b_number_int', 
           'country_a', 'country_b', 'cell_id', 'site_id']
Telcom_ds3.columns = columns
Telcom_ds3.columns
Dfrm1 = pd.concat((Telcom_ds1, Telcom_ds2), axis=0, ignore_index=True)
Dfrm1.head()
DFrm2 = pd.concat([DFrm1, Telcom_ds3], axis=0, ignore_index=False)
Telcom_ds3.drop_duplicates(subset=['cell_id'], inplace=True)
DFrm2.head()

Telcom_Data = pd.concat([DFrm2, Cells_geo], axis=0, ignore_index=False)
Cells_geo.drop_duplicates(subset=['cell_id'], inplace=True)
Telcom_Data.head()
Telcom_Data.date_time = pd.to_datetime(Telcom_Data.datetime)
Telcom_Data['Date'] = Telcom_Data['date_time'].dt.date
Telcom_Data['Time'] = Telcom_Data['date_time'].dt.time
Telcom_Data.dropna(axis=1, how='all')
Telcom_Data.dropna(thresh=5)<jupyter_output><empty_output><jupyter_text>Data Analysis This section will analyse the data to find a solution to the underlying research problem. This will be done by answering selected research questions<jupyter_code>Telcom_Data.head()
# Which were the most used cities in these three days?
Telcom_Data["villes"].value_counts()
# Which cities were used most during home hours?
Busy_Cities = Telcom_Data['villes'], Telcom_Data['Time']
Busy_Cities

Pop_products = Telcom_Data['product'], Telcom_Data['Time']
Telcom_Data['product'].count<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11839755039550906, 'fraction_numerical': 0.029089053329931103, 'mean_word_length': 5.173228346456693, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '597736', 'license': 'Unlicense', 'n_tokens_mistral': 1522, 'n_tokens_neox': 1412, 'n_words': 304, 'path': 'Moringa_Data_Science_Prep_W3_Independent_Project_2020_07_Lilian_Kibathi_Python.ipynb', 'repo_name': 'Nyawira12/IP-WEEK-THREE', 'size': '3919'}