<jupyter_start><jupyter_text>Data Fieldsdatetime - hourly date + timestamp  season -  1 = spring, 2 = summer, 3 = fall, 4 = winterholiday - whether the day is considered a holidayworkingday - whether the day is neither a weekend nor holidayweather - 1: Clear, Few clouds, Partly cloudy, Partly cloudy2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog temp - temperature in Celsiusatemp - "feels like" temperature in Celsiushumidity - relative humiditywindspeed - wind speedcasual - number of non-registered user rentals initiatedregistered - number of registered user rentals initiatedcount - number of total rentals **Workflow**1. Split: train, validation, test2. EDA (...) exploratory data analysis: we look at features, their distributions, cleaning the data, filling missing values, lloking at correlation between features and output and in between features ...etc.3. (We fit a very straightforward simple model as our baseline (e.g. dummy classifier))5. use train and validation data to iteratively improve my model/find best model (feature engineering, hyperparameter tuning, ...)6. apply best model to test data to estimate how the model will perform on new data (using test score, should not vary too much from the best validation score I get in step 4-5)<jupyter_code>df.info()
#all non-null
df.head(10)<jupyter_output><empty_output><jupyter_text>Train Test Split<jupyter_code>X = df.drop(['count'], axis=1) 
y = df['count']

#df.drop(columns='col1') or df.drop('col1', axis=1)
#drop raw. df.drop(index=['row1', 'row2']), df.drop(['row1', 'row2'], axis=0) or df.drop(['row1', 'row2'])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=42)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
train_data = [X_train, y_train]
df = pd.concat(train_data, axis=1)
df<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>#datetime's type is object. We should convert it to datetime.
df.datetime = pd.to_datetime(df.datetime)
#create new fcolumns from 'datetime' olumn
df['year'] = df['datetime'].dt.year
df['month'] = df['datetime'].dt.month
df['day'] = df['datetime'].dt.day
df['hour'] = df['datetime'].dt.hour
df['weekday'] = df['datetime'].dt.day_name()

#dt.day_name, dt.day_name() difference

#pd.concat([df[['day']],pd.get_dummies(df[['year','month','weekday','hour']],columns=['year','month','weekday','hour'])],axis=1)
df['weekday']
#Change the category name for visualization
df["season"] = df.season.map({1: "Spring", 2 : "Summer", 3 : "Fall", 4 :"Winter" })

df["weather"] = df.weather.map({1: " Clear + Few clouds + Partly cloudy + Partly cloudy",\
                                        2 : " Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist ", \
                                        3 : " Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds", \
                                        4 :" Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog " })

#df.season.map
df
dataTypeDf = pd.DataFrame(df.dtypes.value_counts()).reset_index().rename(columns={"index":"variableType",0:"count"})
fig,ax = plt.subplots()
fig.set_size_inches(12,5)
sns.barplot(data=dataTypeDf,x="variableType",y="count",ax=ax)
ax.set(xlabel='variableTypeariable Type', ylabel='Count',title="Variables DataType Count")<jupyter_output><empty_output><jupyter_text>Outliers Analysis<jupyter_code>fig, axes = plt.subplots(nrows=2,ncols=2)
fig.set_size_inches(20, 10)
sns.boxplot(data=df,y="count",orient="v",ax=axes[0][0])
sns.boxplot(data=df,y="count",x="season",orient="v",ax=axes[0][1])
sns.boxplot(data=df,y="count",x="hour",orient="v",ax=axes[1][0])
sns.boxplot(data=df,y="count",x="workingday",orient="v",ax=axes[1][1])

axes[0][0].set(ylabel='Count',title="Box Plot On Count")
axes[0][1].set(xlabel='Season', ylabel='Count',title="Box Plot On Count Across Season")
axes[1][0].set(xlabel='Hour Of The Day', ylabel='Count',title="Box Plot On Count Across Hour Of The Day")
axes[1][1].set(xlabel='Working Day', ylabel='Count',title="Box Plot On Count Across Working Day")
#remove outliers
dfWithoutOutliers = df[np.abs(df['count']-df['count'].mean())<=(2*df['count'].std())] 

#68% of the data falls within one standard deviation of the mean.
#95% of the data falls within two standard deviations of the mean.
#99.7% of the data falls within three standard deviations of the mean.

#understand why np, np.abs later
df.shape, dfWithoutOutliers.shape
fig, axes = plt.subplots(nrows=2,ncols=2)
fig.set_size_inches(20, 10)
sns.boxplot(data=dfWithoutOutliers,y="count",orient="v",ax=axes[0][0])
sns.boxplot(data=dfWithoutOutliers,y="count",x="season",orient="v",ax=axes[0][1])
sns.boxplot(data=dfWithoutOutliers,y="count",x="hour",orient="v",ax=axes[1][0])
sns.boxplot(data=dfWithoutOutliers,y="count",x="workingday",orient="v",ax=axes[1][1])

axes[0][0].set(ylabel='Count',title="Box Plot On Count")
axes[0][1].set(xlabel='Season', ylabel='Count',title="Box Plot On Count Across Season")
axes[1][0].set(xlabel='Hour Of The Day', ylabel='Count',title="Box Plot On Count Across Hour Of The Day")
axes[1][1].set(xlabel='Working Day', ylabel='Count',title="Box Plot On Count Across Working Day")<jupyter_output><empty_output><jupyter_text>Tried two standard deviations and three standard deviation but two looks much less outliers so I use two here.  Correlation Analysis<jupyter_code>corrMat = dfWithoutOutliers[["temp","atemp","casual","registered","humidity","windspeed","count"]].corr()
mask = np.array(corrMat)
mask[np.tril_indices_from(mask)] = False
fig,ax= plt.subplots()
fig.set_size_inches(20,10)
sns.heatmap(corrMat, mask=mask,vmax=.8, square=True,annot=True)
#just visualize
fig,(ax1,ax2,ax3) = plt.subplots(ncols=3)
fig.set_size_inches(15, 5)
sns.regplot(x="temp", y="count", data=dfWithoutOutliers,ax=ax1)
sns.regplot(x="windspeed", y="count", data=dfWithoutOutliers,ax=ax2)
sns.regplot(x="humidity", y="count", data=dfWithoutOutliers,ax=ax3)<jupyter_output><empty_output><jupyter_text>'casual' and 'registered' are not taken into account since they are leakage variables and obviously highly correlated.'atemp' and 'temp' has got strong correlation with each other. I can also guess 'feels like' temprature should be correlated with temperature.'windspeed' is not so correlated with 'count', how every there are many 0 value and if I could deal with it like missing value, I could see this regresson graph minus correlation. And I can imagine if windspeed is fast, people don't wanna use bikes.  Visualizing Distribution Of Data (remove later)<jupyter_code>fig,axes = plt.subplots(ncols=2,nrows=2)
fig.set_size_inches(12, 10)
sns.distplot(df['count'],ax=axes[0][0])
stats.probplot(df['count'], dist='norm', fit=True, plot=axes[0][1])
sns.distplot(np.log(dfWithoutOutliers['count']),ax=axes[1][0])
stats.probplot(np.log1p(dfWithoutOutliers['count']), dist='norm', fit=True, plot=axes[1][1])
dfWithoutOutliers<jupyter_output><empty_output><jupyter_text>Count Vs Month, Season, Hour and Weekday<jupyter_code>fig,(ax1,ax2,ax3,ax4)= plt.subplots(nrows=4)
fig.set_size_inches(20,20)
sortOrder = ["January","February","March","April","May","June","July","August","September","October","November","December"]
hueOrder = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"]

monthAggregated = pd.DataFrame(dfWithoutOutliers.groupby("month")["count"].mean()).reset_index()
monthSorted = monthAggregated.sort_values(by="count",ascending=False) 
sns.pointplot(data=monthSorted,x="month",y="count",ax=ax1,order=sortOrder)
ax1.set(xlabel='Month', ylabel='Avearage Count',title="Average Count By Month")

hourAggregated = pd.DataFrame(dfWithoutOutliers.groupby(["hour","season"],sort=True)["count"].mean()).reset_index()
sns.pointplot(x=hourAggregated["hour"], y=hourAggregated["count"],hue=hourAggregated["season"], data=hourAggregated, join=True,ax=ax2)
ax2.set(xlabel='Hour Of The Day', ylabel='Users Count',title="Average Users Count By Hour Of The Day Across Season",label='big')

hourAggregated = pd.DataFrame(dfWithoutOutliers.groupby(["hour","weekday"],sort=True)["count"].mean()).reset_index()
sns.pointplot(x=hourAggregated["hour"], y=hourAggregated["count"],hue=hourAggregated["weekday"],hue_order=hueOrder, data=hourAggregated, join=True,ax=ax3)
ax3.set(xlabel='Hour Of The Day', ylabel='Users Count',title="Average Users Count By Hour Of The Day Across Weekdays",label='big')

hourTransformed = pd.melt(dfWithoutOutliers[["hour","casual","registered"]], id_vars=['hour'], value_vars=['casual', 'registered'])
hourAggregated = pd.DataFrame(hourTransformed.groupby(["hour","variable"],sort=True)["value"].mean()).reset_index()
sns.pointplot(x=hourAggregated["hour"], y=hourAggregated["value"],hue=hourAggregated["variable"],hue_order=["casual","registered"], data=hourAggregated, join=True,ax=ax4)
ax4.set(xlabel='Hour Of The Day', ylabel='Users Count',title="Average Users Count By Hour Of The Day Across User Type",label='big')

#why month average is not plotted?<jupyter_output><empty_output><jupyter_text>On weekdays more people tend to rent bicycle around 7AM-8AM and 5PM-6PM. Office hour or school hour. On the other hand, 'Saturday' and 'Sunday', more people tend to rent bicycle between 10AM and 4PM.Registered users use in office hours more than casual users.  Dealing with 0 values of 'windspeed' Use mean value to 'windspeed' 0  Creating pipeline<jupyter_code>#colums of previous df are messed up during EDA so crearting again
df = pd.read_csv('train.csv')  #parse_dates=True
df.info()
X = df.drop(['count', 'casual', 'registered'], axis=1) 
y = df['count']
#added 'casual', 'registered because test.csv doesn't have these columns
X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=42)
#df = dfWithoutOutliers
#dfWithoutOutliers = df[np.abs(df['count']-df['count'].mean())<=(2*df['count'].std())] 
def extract(df):
        df.datetime = pd.to_datetime(df.datetime)
        df['day'] = df.datetime.dt.day
        df['hour'] = df.datetime.dt.hour
        df['weekday'] = df.datetime.dt.weekday
        df['year'] = df.datetime.dt.year
        df['month'] = df.datetime.dt.month
        
        return pd.concat([df[['day']],pd.get_dummies(df[['year','month','weekday','hour']],columns=['year','month','weekday','hour'])],axis=1)
preprosessor = ColumnTransformer([
    ('do_nothing', 'passthrough', ['holiday', 'workingday']),
    ('time_extact', FunctionTransformer(extract), ['datetime']),
    ('one_hot_encoding', OneHotEncoder(sparse = False), ['season','weather']),
    ('0_imputer', SimpleImputer(strategy='mean', fill_value=0), ['windspeed']),
    #('bins',KBinsDiscretizer(n_bins= 7, encode = 'onehot-dense', strategy = 'quantile'),['temp','humidity','windspeed']) #R-squared 68 to 70 but RMSLE 9.15 to 9.39...
    ], 
    remainder='drop')  #dropping 'casual', 'registered', 'count', 'atemp', 'date', 'datetime'
# create the model pipeline
pipeline = make_pipeline(preprosessor, LinearRegression())
pipeline.fit(X_train, y_train)
X_train
X_train.info()
pipeline.score(X_train, y_train)
pipeline.score(X_test, y_test)
#Root Mean Squared Logarithmic Error (RMSLE)
def rmsle(y_pred, y,convertExp=True):
    if convertExp:
        y_pred = np.exp(y_pred),
        y = np.exp(y)
    log1 = np.nan_to_num(np.array([np.log(v + 1) for v in y_pred]))
    log2 = np.nan_to_num(np.array([np.log(v + 1) for v in y]))
    calc = (log1 - log2) ** 2
    return np.sqrt(np.mean(calc))
y_pred = pipeline.predict(X_train) 
rmsle(y_pred, y_train, convertExp=True)<jupyter_output>/opt/anaconda3/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: overflow encountered in exp
  result = getattr(ufunc, method)(*inputs, **kwargs)
/var/folders/fs/bmr1lyws1dx3sp07l0gytvmc0000gn/T/ipykernel_81161/3908407482.py:8: RuntimeWarning: overflow encountered in square
  calc = (log1 - log2) ** 2<jupyter_text>Submission...<jupyter_code>kaggle_data = pd.read_csv('test.csv')
kaggle_data
predictions = pipeline.predict(kaggle_data)
submission = pd.DataFrame({'datetime':kaggle_data['datetime'],'count':predictions})
submission
#why minus...<jupyter_output><empty_output><jupyter_text>Simply replace the negative predicted values with something else (e.g. zero). e,g: y_pred[y_pred < 0] = 0.0Scale / transform the target column (bicycle count). Predict the log of count, for example: m.fit(Xtrain, np.log(ytrain)). Of course, then do not forget to "un-log" (i.e. np.exp()) the prediction afterwards, otherwise your model is reporting the log of the demand, which is on a different scale.Use a model that doesn't extrapolate into the negative value problem, e.g. the RandomForestRegressor<jupyter_code>#replace negative count to 0
count_0 = submission['count'].where(submission['count'] >= 0, 0)
submission = pd.DataFrame({'datetime':X_kaggle['datetime'],'count':count_0})
submission
#Convert DataFrame to a csv file that can be uploaded
filename = 'Bike Sharing Demand LR.csv'
submission.to_csv(filename,index=False)
print('Saved file: ' + filename)<jupyter_output>Saved file: Bike Sharing Demand LR.csv<jupyter_text>Memo<jupyter_code>#hour
#day of week
#sin cos, anything has cycle(min, hour)
#how to deal with windspeed


#Outliers Analysis
#Correlation Analysis
    #Feature selection(Lasso, RandomForestRegressor) (features are not so many though)
#Visualizing Count Vs (Month,Season,Hour,Weekday,Usertype)

#Future expansion for liniear regression
#RandomForestRegressor for the model
#Hypeprparameter optimization, , cross validation, gridsearch
#Regularization<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15618173875054608, 'fraction_numerical': 0.016091451871268384, 'mean_word_length': 6.639043381535039, 'pattern_counts': {'":': 1, '<': 46, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '527662', 'license': 'MIT', 'n_tokens_mistral': 4833, 'n_tokens_neox': 4586, 'n_words': 1099, 'path': 'eda.ipynb', 'repo_name': 'strongzero/kaggle-bikeshare', 'size': '13734'}