<jupyter_start><jupyter_text>Cleaning steps There is no missing data (NaN) and datatypes are as expected from examining first fileSecond section appends 6 files (6 months of data from Nov 2011 to April 2021) and outputs a csv fileThe resulting csv file is used for the Tableau visualizations<jupyter_code># Dependencies

import pandas as pd
# create dataframe from csv

df = pd.read_csv("202104-citibike-tripdata.csv")

df
# confirm data types

df.dtypes
# check for missing

df.isnull().sum()
# Check values for suspicious column of gender

df["gender"].value_counts()

# The majority of gender values are 0, which is "unknown"
# Variable will be kept, labeled in Tableau
# Check values for suspicious column of birth year

df["birth year"].value_counts()

# this is likely not a good field - nearly all values are 1970, and the oldest year is 130 years ago
# will keep field but noting concerns<jupyter_output><empty_output><jupyter_text>Append additional months of data<jupyter_code>df_03 = pd.read_csv("202103-citibike-tripdata.csv")

df_03.head(2)
len(df.index)
# add 2021 march data

df_combined = df.append(df_03, sort = False)
len(df_combined.index)
# add 2021 Feb data

df_02 = pd.read_csv("202102-citibike-tripdata.csv")
df_combined = df_combined.append(df_02, sort = False)
len(df_combined.index)
# add 2021 Jan data

df_01 = pd.read_csv("202101-citibike-tripdata.csv")
df_combined = df_combined.append(df_01, sort = False)
len(df_combined.index)
# add 2020 Dec data

df_12 = pd.read_csv("202012-citibike-tripdata.csv")
df_combined = df_combined.append(df_12, sort = False)
len(df_combined.index)
# add 2020 Nov data

df_11 = pd.read_csv("202011-citibike-tripdata.csv")
df_combined = df_combined.append(df_11, sort = False)
len(df_combined.index)
df_combined.head()
# Output csv file

df.to_csv("citibike-merged.csv", encoding='utf-8', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09899417681312864, 'fraction_numerical': 0.05187930121757544, 'mean_word_length': 5.5625, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '547409', 'license': 'MIT', 'n_tokens_mistral': 738, 'n_tokens_neox': 650, 'n_words': 193, 'path': 'Cleaning/cleaning-steps.ipynb', 'repo_name': 'Seltsam1/tableau-visualizations', 'size': '1889'}