<jupyter_start><jupyter_text>Analyzing Ford GoBike Data by <NAME> 1. Preliminary Wrangling> Ford GoBike is a regional public bicycle sharing system located in San Francisco Bay Area, California. In this project, history data of the whole 2018 year was used.<jupyter_code># import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import datetime

import glob
import os

%matplotlib inline<jupyter_output><empty_output><jupyter_text>1.1 Gathering data This dataset is of big size, and separated by 12 months, so I need to combine them in the first step.<jupyter_code>path = r'/Users/zhenghaoxiao/Documents/udacity/communicate_data_findings/bikedata'
files = glob.glob(os.path.join(path, '*.csv'))
df = pd.concat((pd.read_csv(f) for f in files), ignore_index=True)
df.to_csv('bike.csv', index=False)<jupyter_output><empty_output><jupyter_text>1.2 Assessing data<jupyter_code>df = pd.read_csv('bike.csv')<jupyter_output><empty_output><jupyter_text>Now, let's have a glimpse of the dataset:<jupyter_code>df.head()
df.sample(10)<jupyter_output><empty_output><jupyter_text>Check data type of columns:<jupyter_code>df.info(null_counts=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1863721 entries, 0 to 1863720
Data columns (total 16 columns):
duration_sec               1863721 non-null int64
start_time                 1863721 non-null object
end_time                   1863721 non-null object
start_station_id           1851950 non-null float64
start_station_name         1851950 non-null object
start_station_latitude     1863721 non-null float64
start_station_longitude    1863721 non-null float64
end_station_id             1851950 non-null float64
end_station_name           1851950 non-null object
end_station_latitude       1863721 non-null float64
end_station_longitude      1863721 non-null float64
bike_id                    1863721 non-null int64
user_type                  1863721 non-null object
member_birth_year          1753003 non-null float64
member_gender              1753354 non-null object
bike_share_for_all_trip    1863721 non-null object
dtypes: float64(7), int64(2), object(7)
memory usage: 227.5+ MB<jupyter_text>It is obvious that there are some data type problems: timestamp data are not in time date format, ID should be of string format<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>There are 1863721 rows and 16 columns in the original dataset. Duplicated values check:<jupyter_code>df.duplicated().sum()<jupyter_output><empty_output><jupyter_text>Missing values check:<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>Missing values exist in start_station_id, start_station_name, end_station_id, end_station_name, member_birth_year, and member_gender columns. Not all the missing values need to be cleaned. I should just cleaned the missing values in member_birth_year column, because there will be an analysis of interest on member birth year (ages).<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>By assessing the dataset, there are some data quality issues in this dataset:* Missing values exist in **start_station_id, start_station_name, end_station_id, end_station_name, member_birth_year**, and **member_gender** columns* Data type errors:      - should be object: **start_station_id, end_station_id, bike_id**    - should be category: **user_type, member_gender, bike_share_for_all_trip** (notice that is not a real problem, it just makes future usage of those data more convenient)    - should be timestamp: **strat_time, end_time**    - should be int: **member_birth_year*** Unrealistic birth year exists in **member_birth_year*** Data are not sorted by time order  1.3 Cleaning data Make a copy of the original dataset:<jupyter_code>df_cleaned = df.copy()<jupyter_output><empty_output><jupyter_text>Let's first fix the data type problems:* Data type errors:      - should be object: **start_station_id, end_station_id, bike_id**    - should be category: **user_type, member_gender, bike_share_for_all_trip** (notice that is not a real problem, it just makes future usage of those data more convenient)    - should be timestamp: **strat_time, end_time**    **Define**Use pandas astype and to_datetime functions to convert the data type of those columns**Code** Convert timestamps:<jupyter_code>df_cleaned.start_time = pd.to_datetime(df_cleaned.start_time)
df_cleaned.end_time = pd.to_datetime(df_cleaned.end_time)<jupyter_output><empty_output><jupyter_text>Prepare category data:<jupyter_code>df_cleaned.user_type = df_cleaned.user_type.astype('category')
df_cleaned.member_gender = df_cleaned.member_gender.astype('category')
df_cleaned.bike_share_for_all_trip = df_cleaned.bike_share_for_all_trip.astype('category')<jupyter_output><empty_output><jupyter_text>Change IDs to string instead of float:<jupyter_code>df_cleaned.bike_id = df_cleaned.bike_id.astype(str)
df_cleaned.start_station_id = df_cleaned.start_station_id.astype(str)
df_cleaned.end_station_id = df_cleaned.end_station_id.astype(str)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>df_cleaned.head()
df_cleaned.info(null_counts=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1863721 entries, 0 to 1863720
Data columns (total 16 columns):
duration_sec               1863721 non-null int64
start_time                 1863721 non-null datetime64[ns]
end_time                   1863721 non-null datetime64[ns]
start_station_id           1863721 non-null object
start_station_name         1851950 non-null object
start_station_latitude     1863721 non-null float64
start_station_longitude    1863721 non-null float64
end_station_id             1863721 non-null object
end_station_name           1851950 non-null object
end_station_latitude       1863721 non-null float64
end_station_longitude      1863721 non-null float64
bike_id                    1863721 non-null object
user_type                  1863721 non-null category
member_birth_year          1753003 non-null float64
member_gender              1753354 non-null category
bike_share_for_all_trip    1863721 non-null category
dtypes: category(3), datetime64[ns](2), float[...]<jupyter_text>I did not convert the member_birth_year column here because we can do some feature engineering on this column to make it more clear. **Define**Convert the member_birth_year to member_age which can make it more clear for our analysis **Code**<jupyter_code>df_cleaned['member_age'] = 2019 - df_cleaned['member_birth_year']<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>df_cleaned.info(null_counts=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1863721 entries, 0 to 1863720
Data columns (total 17 columns):
duration_sec               1863721 non-null int64
start_time                 1863721 non-null datetime64[ns]
end_time                   1863721 non-null datetime64[ns]
start_station_id           1863721 non-null object
start_station_name         1851950 non-null object
start_station_latitude     1863721 non-null float64
start_station_longitude    1863721 non-null float64
end_station_id             1863721 non-null object
end_station_name           1851950 non-null object
end_station_latitude       1863721 non-null float64
end_station_longitude      1863721 non-null float64
bike_id                    1863721 non-null object
user_type                  1863721 non-null category
member_birth_year          1753003 non-null float64
member_gender              1753354 non-null category
bike_share_for_all_trip    1863721 non-null category
member_age                 1753003 non-null f[...]<jupyter_text>**Define**The problems with the age and birth year column is still existing: data type and outlier values **Code**<jupyter_code>df_cleaned.member_age.sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>There are many values higher than 100, which is abnormal, I need to set a threshold and delete those abnormal values higher than the threshold<jupyter_code>df_cleaned.member_age.describe(percentiles = [.995])<jupyter_output><empty_output><jupyter_text>Since 70 years old will cover 99.5% of the age distribution, and the smallest age is 19, I decide to use 69 years old as the threshold<jupyter_code>df_cleaned = df_cleaned.query('member_age <= 69')<jupyter_output><empty_output><jupyter_text>Now we can convert the data type of those two columns to int<jupyter_code>df_cleaned.member_age = df_cleaned.member_age.astype(int)
df_cleaned.member_birth_year = df_cleaned.member_birth_year.astype(int)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>df_cleaned.member_age.describe()
df_cleaned.info(null_counts=True)
df_cleaned.member_age.plot.hist();<jupyter_output><empty_output><jupyter_text>**Define**Extract the month, day of the week, and hour from the start_time column. Calculate the duration in minutes. This is not a real data quality or tidiness problem, it just makes the later analysis easier. **Code**<jupyter_code>df_cleaned['start_time_month'] = df_cleaned['start_time'].dt.strftime('%B')
df_cleaned['start_time_month_num'] = df_cleaned['start_time'].dt.month.astype(int)
df_cleaned['start_time_weekday'] = df_cleaned['start_time'].dt.strftime('%a')
df_cleaned['start_time_hour'] = df_cleaned['start_time'].dt.hour
df_cleaned['duration_min'] = df_cleaned['duration_sec']/60
df_cleaned['duration_min'] = df_cleaned['duration_min'].astype(int)<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>df_cleaned.head()
df_cleaned.info(null_counts=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1743806 entries, 0 to 1863720
Data columns (total 22 columns):
duration_sec               1743806 non-null int64
start_time                 1743806 non-null datetime64[ns]
end_time                   1743806 non-null datetime64[ns]
start_station_id           1743806 non-null object
start_station_name         1732368 non-null object
start_station_latitude     1743806 non-null float64
start_station_longitude    1743806 non-null float64
end_station_id             1743806 non-null object
end_station_name           1732368 non-null object
end_station_latitude       1743806 non-null float64
end_station_longitude      1743806 non-null float64
bike_id                    1743806 non-null object
user_type                  1743806 non-null category
member_birth_year          1743806 non-null int64
member_gender              1743806 non-null category
bike_share_for_all_trip    1743806 non-null category
member_age                 1743806 non-null int[...]<jupyter_text>**Define*** Data is not sorted by time order, sort the dataset to make it follows the time order: from oldest record to latest record**Code**<jupyter_code>df_cleaned = df_cleaned.sort_values(by='start_time')<jupyter_output><empty_output><jupyter_text>**Test**<jupyter_code>df_cleaned.head()<jupyter_output><empty_output><jupyter_text>Now the dataset follows the time order from oldest record to latest record.  1.4 Storing data<jupyter_code>df_cleaned.to_csv('bike_clean.csv', index=False)<jupyter_output><empty_output><jupyter_text>Read the data again for future analysis:<jupyter_code>df = pd.read_csv('bike_clean.csv')<jupyter_output><empty_output><jupyter_text>What is the structure of your dataset?> The cleaned dataset has 1743806 rows and 22 columns, including ID, name, latitude, longitude of start and end station, start time and data, end time and date, bike ID, user type, member year of birth and member gender.In addition to the original 16 columns, I added 6 new columns derived from feature engineering: **member_age, start_time_month, start_time_month_num, start_time_weekday, start_time_hour, duration_min**. What is/are the main feature(s) of interest in your dataset?> * Time date data can help us check which month of a year, which weekday of a week people tend to use this service, base on that, business decision on promotions can be made to improve the revenue. > * User data: user type, gender and age can be used to figure out which kind of people are willing to pay for this sharing bike product, advertisement strategy can be adjusted according to those findings. What features in the dataset do you think will help support your investigation into your feature(s) of interest?> Trip duration can also be a good help, if we can find the trip duration distribution pattern, we can adjust the pricing strategy, and also predict the average lifetime of a random bicycle.  2. Univariate Exploration The dataset covered the whole year of 2018, it is ordinary that one would like to check the monthly usage of Ford GoBike: **Question*** What is the distribution of monthly usage of the Ford GoBike? **Code**<jupyter_code>from matplotlib import rcParams
rcParams.update({'figure.autolayout': True})

base_color = sns.color_palette()[0]
month_plt = sns.catplot(data=df, x='start_time_month', kind='count', color=base_color, aspect=1.5);
month_plt.set_axis_labels('Month', 'Number of Rides'),
month_plt.fig.suptitle('Ford GoBike Monthly Usage', fontsize=14, fontweight='bold');
month_plt.set_xticklabels(rotation=50);<jupyter_output><empty_output><jupyter_text>* According to the plot, there is a pattern that people tend to ride more frequently during Summer and mid autumn: May to October, in which October is the most popular month; January, February, and March are months with the lowest rides. This should be attributed to the weather and climate of Bay Area. Just like the monthly usage, the weekly usage of the Ford GoBike should also unveil some patterns**Question*** What is the distribution of weekly usage of the Ford GoBike?**Code**<jupyter_code>weekday = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
week_plt = sns.catplot(data=df, x='start_time_weekday', kind='count', color=base_color, order=weekday)
week_plt.set_axis_labels('Weekdays', 'Number of Rides')

week_plt.fig.suptitle('Ford GoBike Weekly Usage', fontsize=14, fontweight='bold');

week_plt.set_xticklabels(rotation=50);<jupyter_output><empty_output><jupyter_text>* This plot shows that people ride bike twice the frequency in weekdays than weekends. It indicates that most people ride Ford GoBike for commuting. What about the hourly usage distribution? **Question*** What is the distribution of hourly usage of the Ford GoBike?**Code**<jupyter_code>hour_plt = sns.catplot(data=df, x='start_time_hour', kind='count', color=base_color, aspect=1.7);

hour_plt.set_axis_labels('Hours', 'Number of Rides')

hour_plt.fig.suptitle('Ford GoBike Hourly Usage', fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* Most rides are occurring on rush hours. It enhanced the hypothesis we framed on the weekly plot: most usage of Ford GoBike is commuting. **Question*** What is the duration distribution of Ford GoBike data?**Code** Let's first decide the bin_edges we'd like to use in the histogram:<jupyter_code>df.duration_min.describe()
bin_edges = np.arange(0, 45, 1)
ticks = [0, 5, 10, 15, 20, 25, 30,  35, 40, 45]
labels = ['{}'.format(val) for val in ticks]

plt.hist(data=df, x='duration_min', bins=bin_edges);

plt.title("Ford GoBike Trip Duration in Minutes", fontsize=14, fontweight='bold')
plt.xlabel('Duration (Min)')
plt.xticks(ticks, labels)
plt.ylabel('Number of Rides');<jupyter_output><empty_output><jupyter_text>* Trip duration data is right skewed with a mean of nearly 10 minutes. It indicates people ride bikes mainly within a short distance. **Question*** What is the distribution of user's age?**Code**<jupyter_code>plt.hist(data=df, x='member_age');
plt.title("Ford GoBike User's Age", fontsize=14, fontweight='bold')
plt.xlabel('Age of Members (Years)')
plt.ylabel('Number of Bike Trips');<jupyter_output><empty_output><jupyter_text>* Majority of the users falls between the age of 25 to 40. The distribution is right skewed. **Question*** What is the users' proportion of different genders?**Code**<jupyter_code>user_gender = df.member_gender.value_counts()
labels=['Male', 'Female', 'Other']
plt.pie(user_gender, labels=labels, startangle=90, 
        autopct= '%1.1f%%', counterclock = False);

plt.title('Ford GoBike User Proportion by Gender', fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* 73.4% of the users are male, while 25.0% of the users are female and 1.5% of the users are other genders. **Question*** What is the users' proportion by user type (customer or subscriber)?**Code**<jupyter_code>user_type = df.user_type.value_counts()
labels=['Subscriber', 'Customer']
plt.pie(user_type, labels=labels, startangle=90, 
        autopct= '%1.1f%%', counterclock = False);

plt.title('Ford GoBike User Proportion by Type', fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>Discuss the distribution(s) of your variable(s) of interest. Were there any unusual points? Did you need to perform any transformations?> There are some general patterns found in the dataset:* people tend to ride more from May to October* most people use the bike for commuting* the average duration per ride is around 10 minutes* the major gender of the users is male* the majority of user type is subscriber Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?> Yes, as the user ages have some unrealistic outliers that higher than 100 years old, to filter those outliers our, I chose a threshold of 69 years old which already covered 99.5% of the user.   3. Bivariate Exploration **Question*** Is the Ford GoBike Monthly Usage different by Gender?**Code**<jupyter_code>g = sns.catplot(data=df, x='start_time_month', col="member_gender", kind='count', 
                sharey=False, color=base_color)
g.set_axis_labels("Month", "Bike Rides")
g.set_titles("{col_name}")
g.set_xticklabels(rotation=50)
g.fig.suptitle('Monthly Usage by Gender', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* Those distributions are nearly the same. People ride more from May to October, no matter which gender they are. **Question*** Is the Ford GoBike Weekly Usage different by gender?**Code**<jupyter_code>weekday = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
g = sns.catplot(data=df, x='start_time_weekday', col="member_gender", kind='count', 
                sharey=False, color=base_color, order=weekday)
g.set_axis_labels("Weekday", "Bike Rides")
g.set_titles("{col_name}")
g.set_xticklabels(rotation=50)
g.fig.suptitle('Weekly Usage by Gender', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* Those distributions are nearly the same. People ride more on weekdays, no matter which gender they are. **Question*** Is the Ford GoBike Monthly Usage different by user type(subscriber and customer)?**Code**<jupyter_code>g = sns.catplot(data=df, x='start_time_month', col="user_type", kind='count', 
                sharey=False, color=base_color)
g.set_axis_labels("Month", "Bike Rides")
g.set_titles("{col_name}")
g.set_xticklabels(rotation=50)
g.fig.suptitle('Monthly Usage by User Type', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* The distributions remain the same. People ride more from May to October no matter which user type they are. So, there is no need to adjust the advertising strategy for different user types. **Question*** Is the Ford GoBike Weekly Usage different by user type(subscriber and customer)?**Code**<jupyter_code>weekday = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
g = sns.catplot(data=df, x='start_time_weekday', col="user_type", kind='count', 
                sharey=False, color=base_color, order=weekday)
g.set_axis_labels("Weekday", "Bike Rides")
g.set_titles("{col_name}")
g.set_xticklabels(rotation=50)
g.fig.suptitle('Weekly Usage by User Type', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* We get some findings here! Compared with subscribers, customers are more likely to ride on weekends. Providing promotions for customers on weekends might be a reasonable way to increasing the revenue. **Question*** Is the Ford GoBike Hourly Usage different by user type(subscriber and customer)?**Code**<jupyter_code>g = sns.catplot(data=df, x='start_time_hour', col="user_type", kind='count', 
                sharey=False, color=base_color)
g.set_axis_labels("Hours", "Bike Rides")
g.set_titles("{col_name}")
g.fig.suptitle('Hourly Usage by User Type', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* Yes, seem like customers ride bike often in between 8 am to 7 pm, while subscribers often ride bike on rush hours. **Question*** Is the duration time per ride different between user types?**Code**<jupyter_code>g = sns.catplot(data=df, y='duration_min', col="user_type", kind='violin', color=base_color)

g.set_titles(col_template = '{col_name}')
g.set_axis_labels("", "Duration (Min)")
g.fig.suptitle('Ford GoBike Duration by User Type', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>These distributions is skewed, I should build this visualization again with data subset of duration_min lower than 60 minutes.<jupyter_code>low_duration = df.query('duration_min < 60')
g = sns.catplot(data=low_duration, y='duration_min', col="user_type", kind='violin', color=base_color)

g.set_titles(col_template = '{col_name}')
g.set_axis_labels("", "Duration (Min)")
g.fig.suptitle('Ford GoBike Duration by User Type', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* On average, customers tend to spend more time per ride compared with subscribers.   Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?> Generally, people tend to ride more in summer and autumn, rush hour, and weekdays. But there are some differences lies between customer usage and subscriber usage: customers tend to have higher duration per ride, customers tend to ride on weekends. Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?> There is some difference of the bike usage between customers and subscribers: customers tend to spend more time per ride, and more likely to ride on weekends instead of weekdays.  4. Multivariate Exploration **Question*** What is the monthly bike usage by user type and then by gender?**Code**<jupyter_code>g = sns.catplot(data=df, x='start_time_month', col="user_type", hue="member_gender", 
                kind='count', sharey=False, legend=False)
g.set_axis_labels("Month", "Bike Rides")
g.set_xticklabels(rotation=50)
g.set_titles("{col_name}")
plt.legend(loc='upper left')
g.fig.suptitle('Monthly Usage Per User Type and Gender', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* The plot shows no much difference compared with the bivariate one, we can say that gender and user type have no interactions on monthly usage of Ford GoBike. There is another point we can see from the plot that customer has higher female proportion than subscribers. **Question*** What is the weekly bike usage by user type and then by gender?**Code**<jupyter_code>weekday = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
g = sns.catplot(data=df, x='start_time_weekday', col="user_type", hue="member_gender", 
                kind='count', sharey=False, legend=False, order=weekday)
g.set_axis_labels("Weekday", "Bike Rides")
g.set_xticklabels(rotation=50)
g.set_titles("{col_name}")
plt.legend(loc='upper left')
g.fig.suptitle('Weekly Usage Per User Type and Gender', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* There is no much difference compared with the bivariate plot. **Question*** What is the hourly bike usage by user type and then by gender?**Code**<jupyter_code>g = sns.catplot(data=df, x='start_time_hour', col="user_type", hue="member_gender", 
                kind='count', sharey=False, legend=False)
g.set_axis_labels("Hours", "Bike Rides")

g.set_titles("{col_name}")
plt.legend(loc='upper left')
g.fig.suptitle('Hourly Usage Per User Type and Gender', y=1.05, fontsize=14, fontweight='bold');<jupyter_output><empty_output><jupyter_text>* No new findings compared with the bivariate plot. **Question*** Since the main different between subscriber and customer is on hours and weekdays, we can build a heat map with weekday and hour to illustrate the different trend of riding Ford GoBike between them.**Code**<jupyter_code>df['start_time_weekday'] = pd.Categorical(df['start_time_weekday'], 
                                          categories=['Mon','Tue','Wed','Thu','Fri','Sat', 'Sun'], 
                                          ordered=True)
plt.figure(figsize=(10, 8))
plt.suptitle('Usage Heatmap for Customers and Subscribers', y=1.05, fontsize=14, fontweight='bold')



# plot for subscriber
plt.subplot(1, 2, 1)
df_subscriber = df.query('user_type == "Subscriber"').groupby(["start_time_hour", "start_time_weekday"])["bike_id"].size().reset_index()
df_subscriber = df_subscriber.pivot("start_time_hour", "start_time_weekday", "bike_id")
sns.heatmap(df_subscriber, cmap="YlGnBu")

plt.title('Subscriber', y=1.01)
plt.xlabel('Weekday')
plt.ylabel('Hour')

# plot for customer
plt.subplot(1, 2, 2)
df_customer = df.query('user_type == "Customer"').groupby(["start_time_hour", "start_time_weekday"])["bike_id"].size().reset_index()
df_customer = df_customer.pivot("start_time_hour", "start_time_weekday", "bike_id")
sns.heatmap(df_customer, cmap="YlGnBu")

plt.title('Customer', y=1.01)
plt.xlabel('Weekday')
plt.ylabel('');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10096172415124947, 'fraction_numerical': 0.03267544706654822, 'mean_word_length': 5.576581153162306, 'pattern_counts': {'":': 0, '<': 184, '<?xml version=': 0, '>': 191, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '490966', 'license': 'MIT', 'n_tokens_mistral': 8796, 'n_tokens_neox': 7993, 'n_words': 2474, 'path': 'communicate_data_findings.ipynb', 'repo_name': 'ZhenghaoXiao32/communicate-data-findings-udacity', 'size': '25898'}