<jupyter_start><jupyter_text>plot distributions - trip_distance, fare_amount, total_amount  distance<jupyter_code># the min of destance is negative .!
# distance 
sns.boxplot(data=df,y='trip_distance')
df.trip_distance.min() , df.trip_distance.max()
df[df["trip_distance"] < 0]<jupyter_output><empty_output><jupyter_text>we have some problems with distance:    - outlier in min and max    - negative numbers    - big std  fare_amount<jupyter_code>sns.boxplot(data=df,y='fare_amount')
df[df["fare_amount"] < 0].count()<jupyter_output><empty_output><jupyter_text>> 3076 have negative numbers .!<jupyter_code>df.fare_amount.median()
sns.distplot(df.fare_amount);


# Add title and axis names
plt.title('amount per trip')
plt.xlabel('amount ')
plt.ylabel('count')

# plot
plt.show()
df.fare_amount.hist(bins=20,range=(0, 70))


# Add title and axis names
plt.title('amount per trip')
plt.xlabel('amount ')
plt.ylabel('count')

# plot
plt.show()<jupyter_output><empty_output><jupyter_text>> most of pepole ranged from 0 to 80 doller<jupyter_code>df[df['fare_amount'] > 55 ].count()
df[df['fare_amount'] < 0 ].count()[1]
df['fare_amount'].count()
df[((df['fare_amount'] > 0)  & (df['fare_amount'] < 50 ))]['fare_amount'].describe()
df[((df['fare_amount'] < 0)  | (df['fare_amount'] > 55 ))].count()<jupyter_output><empty_output><jupyter_text>> we can take all the out of this range as outlier  time<jupyter_code>time_expand =  df.tpep_dropoff_datetime -  df.tpep_pickup_datetime
time_expand = time_expand.dt.total_seconds()
time_expand.describe()
time_expand.hist(bins=20,range=(0, 5000))


# Add title and axis names
plt.title('time by minutes in each trip')
plt.xlabel('minutes ')
plt.ylabel('count')

# plot
plt.show()<jupyter_output><empty_output><jupyter_text>> big standard deveation > we need to treat with outlier again<jupyter_code># fare_amount >> we will reuren to it leter<jupyter_output><empty_output><jupyter_text>correlation<jupyter_code># add time expand in the datafram
df["trip_time"] = time_expand
sns.heatmap(df.corr())
plt.figure(figsize=(16, 6))
# define the mask to set the values in the upper triangle to True
mask = np.triu(np.ones_like(df.corr(), dtype=np.bool_))
heatmap = sns.heatmap(df.corr(), mask=mask, vmin=-1, vmax=1, annot=True, cmap='BrBG')
heatmap.set_title('Triangle Correlation Heatmap', fontdict={'fontsize':18}, pad=16);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1535234899328859, 'fraction_numerical': 0.016778523489932886, 'mean_word_length': 7.03030303030303, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '527764', 'license': 'FTL', 'n_tokens_mistral': 944, 'n_tokens_neox': 873, 'n_words': 182, 'path': 'notebooks/02_analytics.ipynb', 'repo_name': 'tarekyehya/NYC-Taxi-Trips', 'size': '2384'}