<jupyter_start><jupyter_text>PyPI Data Analysis (22nd July) This notebook looks at the data collected about each project uploaded on PyPI - the names, dates of release, versions, and descriptions - and performs an EDA (Exploratory Data Analysis) after narrowing down the data.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Orginal Data<jupyter_code>df = pd.read_csv('cleaned_project_and_date.csv')
df = df.drop("Unnamed: 0", axis = 1)
df.head()<jupyter_output><empty_output><jupyter_text>Selecting Required Data<jupyter_code>useful_df = pd.DataFrame()
useful_df["Project Name"] = df.name
useful_df["Release Date"] = pd.to_datetime(df.old_date)
useful_df = useful_df.sort_values(by = "Release Date")
useful_df = useful_df.reset_index()
useful_df = useful_df.groupby("Release Date").agg({"Project Name": "count"})
useful_df.head()<jupyter_output><empty_output><jupyter_text>Final Data<jupyter_code>range_df = pd.DataFrame()
range_df["Release Date"] = pd.Series(pd.date_range(start = "2002-11-05", \
        end = "2019-07-18", freq = "d"))
final = pd.merge(range_df, useful_df, on = "Release Date", how = "left")
final = final.fillna(0)
final["On Day"] = final["Project Name"]
final = final.drop("Project Name", axis = 1)
final = final.set_index("Release Date")
temp = final.cumsum(axis = 0)
final["So Far"] = temp["On Day"]
final.head(10)<jupyter_output><empty_output><jupyter_text>Trends with number of projects released per day This graph plots the number of projects released everyday. With a few exceptional days, the general trend is that the number of projects released per day slowly increases with time.<jupyter_code>final["On Day"].plot()
plt.ylabel('Number of Projects')<jupyter_output><empty_output><jupyter_text>The next two graphs show the number of projects released per day year-wise, for 2017 and 2016. It is hard to see any trends in 2017 due to outliers, but it is a lot more visible in 2016 - there seems to be some sort of seasonality trend.<jupyter_code>ax = final.loc["2017", "On Day"].plot()
ax.set_label("Number")
ax = final.loc["2016", "On Day"].plot()
ax.set_label("Number")<jupyter_output><empty_output><jupyter_text>The next two graphs take 2 randomly chosen months and try to analyse the seasonality seen above. It's clear that there is a similar trend each week - the number of projects released over the weekends is low, but during the weekdays is high.<jupyter_code>ax = final.loc["2017-01":"2017-02", "On Day"].plot(marker = 'o', linestyle = '-')
ax.set_ylabel("Number")
ax = final.loc["2016-05":"2016-06", "On Day"].plot(marker = 'o', linestyle = '-')
ax.set_ylabel("Number")<jupyter_output><empty_output><jupyter_text>The next two box plots show the mean number of projects released per day, grouped by the day of the week. The first box plot is with outliers, the second without. It confirms that the number of projects released over the weekends is the least.<jupyter_code>fig, axes = plt.subplots(figsize = (12, 8), sharex = True)
for name, ax in zip(["On Day"], [axes]):
    sns.boxplot(data = final, x = final.index.weekday_name, y = name, ax = ax)
ax.set_ylabel("Number")
ax.set_title(name)
temp_df = final[final["On Day"] < 350]
fig, axes = plt.subplots(figsize = (12, 8), sharex = True)
for name, ax in zip(["On Day"], [axes]):
    sns.boxplot(data = temp_df, x = temp_df.index.weekday_name, y = name, ax = ax)
ax.set_ylabel("Number")
ax.set_title(name)<jupyter_output><empty_output><jupyter_text>The next two box plots show the mean number of projects released per day, grouped by the month of the year. The first box plot is with outliers, the second without. There does not seem to be a specific trend here.<jupyter_code>fig, axes = plt.subplots(figsize = (12, 8), sharex = True)
for name, ax in zip(["On Day"], [axes]):
    sns.boxplot(data = final, x = final.index.month, y = name, ax = ax)
ax.set_ylabel("Number")
ax.set_title(name)
temp_df = final[final["On Day"] < 350]
fig, axes = plt.subplots(figsize = (12, 8), sharex = True)
for name, ax in zip(["On Day"], [axes]):
    sns.boxplot(data = temp_df, x = temp_df.index.month, y = name, ax = ax)
ax.set_ylabel("Number")
ax.set_title(name)<jupyter_output><empty_output><jupyter_text>Trends with total number of projects released This graph shows the relationship beteween the release date and the total number of projects released so far. The graph is polynomial/exponential in nature.<jupyter_code>final["So Far"].plot()
plt.ylabel("Number of projects")<jupyter_output><empty_output><jupyter_text>Release in 2019 The next two graphs show the trend of total number of projects released, and the number of projects released per day, with the release date.<jupyter_code>ax = final.loc["2019", "On Day"].plot()
ax.set_ylabel("Number")
ax = final.loc["2019", "So Far"].plot()
ax.set_ylabel("Number")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1264461132535011, 'fraction_numerical': 0.021108179419525065, 'mean_word_length': 5.750684931506849, 'pattern_counts': {'":': 3, '<': 47, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '645262', 'license': 'MIT', 'n_tokens_mistral': 1730, 'n_tokens_neox': 1607, 'n_words': 529, 'path': 'PyPI Data Analysis (22nd July).ipynb', 'repo_name': 'mihikabairathi/PyPI-Package-Prediction', 'size': '4927'}