<jupyter_start><jupyter_text>![title](edel.JPG)  Edelweiss Hackathon-Machine Learning Predict Price Movement Broad ObjectiveTo predict short term (intraday) price movement of various stocks on the basis of its price and combination of multiple features provided. Data Provided:We have provided historical 5-min tick data for a universe of more than 150 stocks. The data has the following features:* **Mid_Price:** The price over which the returns need to be calculated* **Date and Time*** **46 anonymized features** MethodologyThe algorithm will decide whether to take a position in a stock (buy/sell)Returns calculated will be calculated as* If there is no current position in any stocks  at any point;  the algorithm can either buy an stocks or sell an stocks ; If the stock is bought it’s called a long position, and if the stock is sold its called a short position* If there is a long position, the algo can only square-off the position: It means it will sell the current long position and become position neutral.* If there is a short position, the algo can only square-off the position: It means it will buy  the current short position and become position neutral.* At 3:15 PM every day the algo mandatorily should square-off any open position.* Return should be calculated only when the positions are square-off: If the position was long the return calculated will be equal to ((exit price- entryprice)/entryprice)* If the position was short the return calculated will be equal to ((entryprice -exit price)/entryprice)* For any new entry with exit the “number of trades” will be equal to 1; for example if one enters underlying4 at 1:00 PM and exits at 3:15 , number of the trade for that day for that particular stock will be 1.* Since all the price prediction needs to be done intraday , all the positions can get opened after (9:30 AM) and should exit at or before (3:15 PM) Output:It should have the following information* Stocks wise daily return (daily return(a)):  Return for any day for any stock : sum of all trades return for that day: For example : for underlying 7 : the algo enters long at 1:20 PM and square-off the position at 2:40 PM: and again goes short 2:45 and square-off at 3:15. The return for that day will be the return of two trades taken that day.* Stocks wise number of daily trades (trades(b)) : Trades for any day for any stock : sum of all trades for that day. For example: the trades will be equal to 2, as in the above mentioned example. Detailed Objective:* You have predict the Mid-price on the testing data and then build an algorithm that predicts the Buy/Sell decisions (Long/ Short positions) for the predicted Mid-prices.* The final optimisation problem would be to maximise  of ((a)-(0.12%)*(b)) *(b), this will be the criterion for selecting the price predicting algosHowever, there must be a minimum of 20 trades per stocks; averaged across stock.  Predicting MidPrice<jupyter_code># This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

import os
print(os.listdir("../input/predict-price-movement-kernel-1"))

TRAIN_PATH='../input/hackerearth-predict-price-movement-edlewiess/dataset/dataset/train/'
TEST_PATH='../input/hackerearth-predict-price-movement-edlewiess/dataset/dataset/test/'

# Any results you write to the current directory are saved as output.
train1=pd.read_csv(TRAIN_PATH+'train1.csv')
test1=pd.read_csv(TEST_PATH+'test1.csv')
for i in [2,3]:
    train1=train1.append(pd.read_csv(TRAIN_PATH+'train'+str(i)+'.csv'))
    test1=test1.append(pd.read_csv(TEST_PATH+'test'+str(i)+'.csv'))
import matplotlib.pyplot as plt
%matplotlib inline
# test1.head()
# plt.figure(figsize=(18,10))
# strain[strain['Symbol']=='und_1']['MidPrice'].plot()
# plt.axhline(y=0, color='r')
# stest[stest['Symbol']=='und_1']['MidPrice'].plot()

plt.figure(figsize=(18,10))
train1[train1['Symbol']=='und_101']['MidPrice'].plot()

plt.figure(figsize=(18,10))
train1[train1['Symbol']=='und_109']['MidPrice'].plot()
# train[train['Symbol']=='und_109']

# plt.plot(x=pd.to_datetime(train[train['Symbol']=='und_76']['UID1']))
# train[train[train['Date']=='04-01-2015']['Symbol']=='und_76']
# train[train['Symbol']=='und_109']
plt.figure(figsize=(18,10))
train1[(train1['Date']=='04-01-2015') & (train1['Symbol']=='und_109')].tail()

# train[train['Symbol']=='und_109']
plt.figure(figsize=(18,10))
train1[train1['Symbol']=='und_159']['MidPrice'].plot()<jupyter_output><empty_output><jupyter_text>Check how many have negative values<jupyter_code>f=[]
for j in train1.columns:
    if j.startswith('Feature') and j not in ['Feature42']:
        print(j,train1[train1[j]<0].shape)
        if train1[train1[j]<0].shape[0]>240676:
            f.append(j)
print(f)
train1.head()
import seaborn as sns
plt.figure(figsize=(12,10))
sns.heatmap(train1.corr())
train=train1.copy()
test=test1.copy()<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>from sklearn.preprocessing import LabelEncoder
l=LabelEncoder()

train['Symbol']=l.fit_transform(train['Symbol'])
test['Symbol']=l.transform(test['Symbol'])
train['mDate']=pd.to_datetime(train['Date']+' '+train['Time'],format='%d-%m-%Y %H:%M')
def ismorning(val):
    if val>='12:00':
        return 1
    else:
        return 0
train['d']=train['mDate'].dt.day.astype(np.float64)
train['month']=train['mDate'].dt.month.astype(np.float64)
train['week']=train['mDate'].dt.week.astype(np.float64)
train['min']=train['mDate'].dt.minute.astype(np.float64)
train['hr']=train['mDate'].dt.hour.astype(np.float64)
train['weekday']=train['mDate'].dt.weekday.astype(np.float64)
train['ismorning']=train['Time'].apply(ismorning)
train['ismonthstart']=train['mDate'].dt.is_month_start
train['ismonthend']=train['mDate'].dt.is_month_end

test['mDate']=pd.to_datetime(test['Date']+' '+test['Time'],format='%d-%m-%Y %H:%M')
test['d']=test['mDate'].dt.day.astype(np.float64)
test['month']=test['mDate'].dt.month.astype(np.float64)
test['week']=test['mDate'].dt.week.astype(np.float64)
test['min']=test['mDate'].dt.minute.astype(np.float64)
test['hr']=test['mDate'].dt.hour.astype(np.float64)
test['weekday']=test['mDate'].dt.weekday.astype(np.float64)
test['ismorning']=test['Time'].apply(ismorning)
test['ismonthstart']=test['mDate'].dt.is_month_start
test['ismonthend']=test['mDate'].dt.is_month_end
print(train.columns)
# r_feat=['UID','Date','mDate','Time','Feature42','Feature30','Feature31','Feature32','Feature33','Feature43','Feature9', 'Feature10',
#         'Feature11', 'Feature12', 'Feature13', 'Feature14', 'Feature15', 'Feature16', 'Feature19', 'Feature20', 'Feature21', 'Feature26', 'Feature27', 'Feature28', 'Feature29']
# r_feat=['UID','Date','mDate','Time','Feature42','Feature30','Feature31','Feature32','Feature33','Feature43']
r_feat=['UID','Date','mDate','Time','Feature42','Feature30','Feature31','Feature32','Feature33','Feature43','Feature9','Feature10',  'Feature11', 'Feature12',
        'Feature13', 'Feature14', 'Feature15', 'Feature16', 'Feature19', 'Feature20', 'Feature21', 'Feature26', 'Feature27', 'Feature28', 'Feature29']

df=train.drop(r_feat,axis=1)
dftest=test.drop(r_feat,axis=1)
X,y=df.drop('MidPrice',axis=1),df['MidPrice']
dftest.columns
X.shape

df.head()
plt.figure(figsize=(12,10))
sns.heatmap(df.corr())

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score,f1_score,mean_squared_error,mean_squared_log_error
# y=train['Fees']
X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.25,random_state = 1994)

categorical_features_indices = np.where(X_train.dtypes =='object')[0]
categorical_features_indices
# X_train.isnull().sum()
from sklearn.ensemble import RandomForestRegressor
# m=RandomForestRegressor(n_estimators=100,random_state=1994)
# m.fit(X_train,y_train)
# pr=m.predict(X_val)
# # m=Rid
# # print(np.sqrt(mean_squared_log_error(np.expm1(y_val.values),np.expm1(p))))
# print(np.sqrt(mean_squared_error(y_val.values,pr)))<jupyter_output><empty_output><jupyter_text>Evaluation and Prediction<jupyter_code>from lightgbm import LGBMRegressor
m=LGBMRegressor(n_estimators=1000,random_state=1994)
# m=RidgeCV(cv=4)
# X_traina,y_traina=augment(X_train.values, y_train.values)
# X_vala,y_vala=augment(X_val.values, y_val.values)
m.fit(X_train,y_train,eval_set=[(X_val,y_val)],eval_metric=['l1','l2','mape'],verbose=100,early_stopping_rounds=100,categorical_feature=[0,-1,-2,-3,-4,-5,-6,-7,-8,-9])
p=m.predict(X_val)
# m=Rid
# print(np.sqrt(mean_squared_log_error(np.expm1(y_val.values),np.expm1(p))))
print(np.sqrt(mean_squared_error(y_val.values,p)))
# print(rmsle_lgb(y_val.values,p))
# X=X.reset_index()
# X=X.drop('index',axis=1)
# X

# X=X.reset_index()
# X=X.drop('index',axis=1)
# X
# dftestX_tr, y_tr = augment(X_train.values, y_train.values)
err=[]
y_pred_tot=[]
i=0
from sklearn.model_selection import KFold,StratifiedKFold
fold=KFold(n_splits=25,shuffle=True,random_state=1994)
for train_index, test_index in fold.split(X,y):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    rf=LGBMRegressor(n_estimators=10000,random_state=1994)
    rf.fit(X_train,y_train,eval_set=[(X_test, y_test)],eval_metric=['l1','l2','mape'],
         verbose=200,categorical_feature=[0,-1,-2,-3,-4,-5,-6,-7,-8,-9]
         , early_stopping_rounds=100
          )
    prc=rf.predict(X_test)
    print("err: ",np.sqrt(mean_squared_error(y_test,prc)))
    err.append(np.sqrt(mean_squared_error(y_test,prc)))
    p = rf.predict(dftest)
    i+=1
    y_pred_tot.append(p)
y_pred_tot
s=pd.DataFrame({'UID':test['UID'],'MidPrice':np.mean(y_pred_tot,0)})
s.head()

s[s['MidPrice']<0].sort_values('MidPrice')
s.to_csv('HE_Edel_pricelgbm.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15936763550667715, 'fraction_numerical': 0.03642969363707777, 'mean_word_length': 6.780748663101604, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '469058', 'license': 'Apache-2.0', 'n_tokens_mistral': 3774, 'n_tokens_neox': 3513, 'n_words': 802, 'path': 'Predict Price Movement/HE- Predict Price Movement - Prices Prediction.ipynb', 'repo_name': 'rajat5ranjan/Edelweiss-Hackathon-Machine-Learning-Hackerearth', 'size': '10184'}