<jupyter_start><jupyter_text>Backtest EngineInstance Variable:* capital_base* open_positions* context object - to store all necessary variables for that algorithm 1. Create a performance dataframe per algo (eg. per stock if it's LGB individual stock) 2. To have an order_target_percent functionFor each order* update instance variables: cash amount* buy price: use open price* append/pop position to open_positions dictionary (symbol, num_shares, date_buy, buy_price, date_sell, target_up, target_low) 3. A metrics report functioninput: performance dataframe output: tables, plots* to create tear sheet for Sharp ratio, Annual volatility, Annual return, cumulative returns(of the full test period)* output plots of equilty curve (porfolio value) or cumulative return* porfolio value rebased compared to HSI* Distribution of monthly returns (histogram)* drawdown periods 4. A performance dataframe functioninput: result dataframe output: performance dataframe* algo_volatility50, return_daily, sharpe ratio* positions (as a dict), orders (a dict or a list)* start_cash, end_cash, pnl_daily (difference in porfolio value compared to previous day)* start_stock_value(as of open price), end_stock_value (as of close price)* exposure* max drawdown50* return_daily, return_daily_annualized Backtest Algorithm - one function for each algoinput: prices and features datasetoutput: performance dataframeMachine Learning Dynamic target (Long only) -Update start cash to performance dataframe1. Iterate through each day of test period2. Iterate through open_positions dictionary    * If up target is met (that day's high is higher than up target) - place sell order    * If low target is met  (that day's low is lower than low target) - place sell order    * If date sell is met - place sell order     3. Split and handle data as needed. Train model and tune hyperparameters. Then predict on test data using best model. If predicted +1    * if the stock is not in open_positions dictionary:        * Place buy order      else:        * Don't do anything        4. Append to a result dataframe of positions, start cash, end cash, porfolio value* porfolio value    * Sumation for all stocks: (close price of that stock * num_share)    * cash5. After all iterations, pass in result dataframe to performance function. Return the performance dataframe Instance Variable:* capital_base* open_positions* context object - to store all necessary variables for that algorithm<jupyter_code>import info
import pandas as pd
from pandas import DataFrame
import numpy as np
import datetime as dt
from datetime import datetime
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score, accuracy_score
from sklearn.ensemble import RandomForestClassifier
import lightgbm as lgb
import warnings
import os.path
#import lightgbm as lgb
from utils import *
from backtest_algos import *

warnings.filterwarnings("ignore")

path_pc = 'C:/Users/admin/Desktop/AI Plan/Finance with AI/Notebooks/'
today = datetime.today()
print("Today's date:", today.strftime("%Y-%m-%d"))
#outcomes_new = load_latest(today, 'outcomes_new_features_', path_pc)
outcomes_new = load_csv(path_pc, 'outcomes_new_features_2020-04-17.csv')
#outcomes_new = load_csv(path_pc, 'outcomes_new_features_dropna.csv')
#outcomes_new = load_csv(path_pc, 'outcomes_new_features_2020-04-09.csv')
list(outcomes_new.columns)
for lookback in [5,10,20,30,40]:
    zscore = lambda x: (x - x.rolling(window=lookback, min_periods=int(lookback*.8)).mean())/x.rolling(window=lookback, min_periods=int(lookback*.8)).std()
    col = 'close'
    outcomes_new[col+'_scaled'+str(lookback)] = outcomes_new.groupby(level='symbol')[col].apply(zscore)<jupyter_output><empty_output><jupyter_text>Result DataFrame* Index = date* columns = porfolio value, orders, positions (as dict)  Change algo to include multiple symbols:* Use last date of outcomes_new_dropna as latest test date* backcount from there and get a list of test dates* For each test date:    * At Open, buy stocks from buy list.        * Calculate best money allocation, with current cash amount    * At close, sell stocks that hit day limit    * Loop thru list of symbols:        * Slice outcomes_new_dropna so that it only contain that symbol        * Get list of trade dates from one-symbol dataframe             * Check if current test date is in this symbol's trade date list. If not, it means this symbol is not trade on                    that date. SKIPP THIS SYMBOL        * From that list calculate training start, valid start, etc.        * Train and make prediction. Add predictions to stock buy list def backtest_rf(outcomes_new):        outcomes_new_dropna = outcomes_new.dropna()        trade_date_list = sorted(list(set(outcomes_new.index.get_level_values('date'))))        features_selected = info.features_lgb_v2  + ['close_scaled5','close_scaled10','close_scaled20','close_scaled30','close_scaled40']        max_impact_pct = 0.001    max_impact_volatility = 'price_chg_1_ema_std50'    total_budget = context.capital_base * 1.2        test_period = 250    test_size = 1    valid_size = 1    training_size = 2000    valid_test_gap = 4    label = 'label_v2'    make a list of test dates according to test period, by back counting from the latest date with target    test_dates = sorted(list(set(outcomes_new_dropna.index.get_level_values('date'))))[-test_period:]        test_dates_converted = []    for ts in test_dates:        test_dates_converted.append(ts.to_pydatetime())        results = pd.DataFrame()        context.predictions = pd.DataFrame()        for test_date in test_dates_converted:                print()                price_list = get_price_list(outcomes_new, test_date)                price_list_open = get_price_list_open(outcomes_new, test_date)                update positions value according to open prices        context.positions_value = positions_value(context.positions, price_list_open)                if 'queue' not in globals() and 'queue' not in locals():            queue = pd.DataFrame()                context.max_positions = int((context.positions_value+context.cash)/(context.typical_amount * \                                        (context.margin_ratio_long+context.margin_ratio_short)/2))        context.max_daily_orders = int(context.max_positions/context.average_holding_period)        context.max_daily_margin_limit = (context.positions_value+context.cash)/context.average_holding_period                remaining_orders = context.max_daily_orders        remaining_margin_limit = context.max_daily_margin_limit                if len(queue)>0:            while True:                if len(queue)==0 or remaining_orders==0 or remaining_margin_limit<=0:                    print ('Break with len(queue) = {}, remaining_orders = {} and remaining_margin_limit = {}'.format(len(queue),                                                                                                                          remaining_orders,                                                                                                                          remaining_margin_limit))                    break                order_list = list(queue[:min(remaining_orders, len(queue))]['symbol'])                yesterday = trade_date_list[trade_date_list.index(test_date)-1]                weights = inverse_volatility_weights(order_list, yesterday, outcomes_new)                                place order                symbol = queue[:1]['symbol'].values[0]                order_type = queue[:1]['action'].values[0]                order_id = ''                num_shares = 1                date_open = test_date                traded_price = price_list_open[:,symbol].values[0]                date_limit = trade_date_list[trade_date_list.index(date_open)+4]                target_up = outcomes_new.loc[(date_open,symbol)]['target_upper_v2']                target_down = outcomes_new.loc[(date_open,symbol)]['target_lower_v2']                percentage = weights[symbol] if len(order_list)>1 else 1                print()                print("[{}] Symbol: {} Date open: {}, Date limit: {}, Traded price: {}.".format(order_type,                                                                                                symbol,                                                                                                date_open.strftime("%Y-%m-%d"),                                                                                                date_limit.strftime("%Y-%m-%d"),                                                                                                traded_price))                order = Order(order_type, symbol, num_shares, date_open, traded_price, date_limit, target_up, target_down)                status, margin_change = order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)                if status == 'successful':                    remaining_orders-=1                    queue=queue.iloc[1:]                    remaining_margin_limit-=margin_change                                    elif status == 'margin_overshoot':                    break                                        elif status == 'min_charge_not_hitting':                    Cancel order                    queue=queue.iloc[1:]        listen to market price of all stocks in positions        If hitting target price, then sell        for i, row in context.positions.iterrows():            if outcomes_new.loc[(test_date,row['symbol'])]['high']>row['target_up']:                order_type = 'CLOSE'                order_id = row['order_id']                num_lots = 1                num_shares = float('Nan')                date_open = float('Nan')                traded_price = row['target_up']                date_limit = float('Nan')                target_up = float('Nan')                target_down = float('Nan')                percentage = 0                                print()                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)                continue            if outcomes_new.loc[(test_date,row['symbol'])]['low']<row['target_down']:                order_type = 'CLOSE'                order_id = row['order_id']                num_lots = 1                num_shares = float('Nan')                date_open = float('Nan')                traded_price = row['target_down']                date_limit = float('Nan')                target_up = float('Nan')                target_down = float('Nan')                percentage = 0                                print()                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)                continue                Near market close - check what stock in position, with date limit = today. Sell it at close price        for i, row in context.positions.iterrows():            print('type of row[date_limit]', type(row['date_limit']))            if test_date >= row['date_limit']:                order_type = 'CLOSE'                order_id = row['order_id']                num_lots = 1                num_shares = float('Nan')                date_open = float('Nan')                traded_price = price_list[:,row['symbol']].values[0]                date_limit = float('Nan')                target_up = float('Nan')                target_down = float('Nan')                percentage = 0                                print()                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)                        After all the buy and sell (imagine the time now is after market close)        Update porfolio value        context.positions_value = positions_value(context.positions, price_list)                if 'symbol' in context.positions.columns:            positions_list = list(context.positions['symbol'])        else:            positions_list = []                                          result = {'positions': positions_list, context.positions.to_dict('records'),                'date': test_date,                 'positions_value': context.positions_value,                 'cash_value': context.cash,                 'porfolio_value': context.positions_value+context.cash}                results = results.append(result, ignore_index=True)                Update margin value due to the closing positions and STOCK VALUE CHANGE. USE CLOSE VALUE FOR THE DAY        context.current_margin = positions_value_absolute(context.positions, price_list) * context.margin_ratio_long                CHECK MAINTENANCE MARGIN        if context.positions_value+context.cash <= context.current_margin*0.8:            print ('ELV lower than!!! MAINTENANCE MARGIN. GAME OVER LIAO.')            return results        End of Day SMA calculation: ELV-Reg T Margin >= 0, cash>=-0.5*stock_value        if context.positions_value+context.cash < context.positions_value * 0.5:            print ('SMA smaller than zero! GAME OVER.')            return results                HERE's the PREDICTION PART        print('Test date: ', test_date)        get all trading dates from dataset of this symbol        data_dates = sorted(list(set(outcomes_new_dropna.index.get_level_values(0))))        data_converted_dates = []        for ts in data_dates:            data_converted_dates.append(ts.to_pydatetime())        Calculate training start date and valid start date by back counting        start_date = data_converted_dates[data_converted_dates.index(test_date)-(valid_size + training_size + valid_test_gap + test_size)]        start_date_valid = data_converted_dates[data_converted_dates.index(test_date)-(valid_size + valid_test_gap + test_size)]        end_date_valid = data_converted_dates[data_converted_dates.index(test_date)-valid_test_gap-test_size]        start_date_test = data_converted_dates[data_converted_dates.index(test_date)-test_size]        end_date_test = test_date        X_y_train, X_y_valid, X_y_test = train_valid_test_split(outcomes_new_dropna, start_date,                                                                    start_date_valid, end_date_valid, start_date_test, end_date_test)        downsample the training set so each class has same number of samples        X_y_train_resampled = downsample_3class(X_y_train, label, 42)        rename all -1 labels to 2 as LGB only takes 0,1,2...        X_y_train_resampled.loc[X_y_train_resampled[label] == -1, label] = 2        X_y_valid.loc[X_y_valid[label] == -1, label] = 2        X_y_test.loc[X_y_test[label] == -1, label] = 2        split into features and target sets        X_train, y_train = feature_target_split(X_y_train_resampled, features_selected, label)        X_valid, y_valid = feature_target_split(X_y_valid, features_selected, label)        X_test, y_test = feature_target_split(X_y_test, features_selected, label)        model_rf =RandomForestClassifier(random_state=42, min_samples_split=8,                                      min_samples_leaf=2, max_features=0.4780494942262089,                                      n_estimators=172, max_leaf_nodes=1000, bootstrap=True, n_jobs=4)        model_rf.fit(X_train, y_train)        y_class_pred = model_rf.predict(X_test)        y_class_pred = [pred.argmax() for pred in y_pred]        X_test['pred'] = y_class_pred            buy_df = pd.DataFrame()        short_df = pd.DataFrame()                                search criteria for coming up with buy/short list        search_criteria = 'momentum_50'        momentum_filter = 'close_scaled5'                                for i, row in X_test.iterrows():            if row['pred'] == 1:                entry = {'symbol': i[1],                        'volatility': row['past_return_5_std50'],                        search_criteria: row[search_criteria]}                buy_df=buy_df.append(entry, ignore_index=True)                            elif row['pred'] == 2:  and row[momentum_filter]<0:                entry = {'symbol': i[1],                        'volatility': row['past_return_5_std50'],                       search_criteria: row[search_criteria]}                short_df=short_df.append(entry, ignore_index=True)                        if len(buy_df)!=0 or len(short_df)!=0:            positive_pct = len(buy_df)/(len(buy_df)+len(short_df))            negative_pct = len(short_df)/(len(buy_df)+len(short_df))        else:            positive_pct = 0            negative_pct = 0                print('Postive percentage: ', positive_pct)        print('Negative percentage: ', negative_pct)        print()                if len(buy_df)>0:            positive_list = list(buy_df['symbol'])        else:            positive_list = []                    if len(short_df)>0:            negative_list = list(short_df['symbol'])        else:            negative_list = []                    predictions_entry = {'date':test_date,                            'num_predictions_long':len(buy_df),                            'num_predictions_short':len(short_df),                            'predictions_long':positive_list,                            'predictions_short':negative_list,                            'percentage_long':positive_pct,                            'percentage_short':negative_pct}                context.predictions=context.predictions.append(predictions_entry, ignore_index=True)                buy_drop_indexes = []        short_drop_indexes = []                for i, row in buy_df.iterrows():            if len(context.positions)>0:                if row['symbol'] in list(context.positions['symbol']):                    buy_drop_indexes.append(i)        for i, row in short_df.iterrows():            if len(context.positions)>0:                if row['symbol'] in list(context.positions['symbol']):                    short_drop_indexes.append(i)                buy_df=buy_df.drop(buy_drop_indexes)        short_df=short_df.drop(short_drop_indexes)                search_criteria = 'volatility'                buy_list = []        short_list = []        if len(buy_df)>0:            buy_df = buy_df.sort_values(by=[search_criteria], ascending=True)            buy_list = list(buy_df['symbol'])                if len(short_df)>0:            short_df = short_df.sort_values(by=[search_criteria], ascending=True)            short_list = list(short_df['symbol'])                    print("--------------------------------")        print('Buy_list symbols:')        print(buy_list)        print('Short_list symbols:')        print(short_list)        print("--------------------------------")                len_buy = len(buy_list)        len_short = len(short_list)                queue = pd.DataFrame()                arrange buy and short list in a queue, alternately buy and short        for symbol_to_buy, symbol_to_short in zip(buy_list, short_list):            queue = queue.append({'symbol':symbol_to_buy,                                  'action':'BUY'}, ignore_index=True)            queue = queue.append({'symbol':symbol_to_short,                                  'action':'SHORT'}, ignore_index=True)        if len_buy>len_short:            for k in range(len_short, len_buy):                queue = queue.append({'symbol':buy_list[k],                                      'action':'BUY'}, ignore_index=True)        elif len_short>len_buy:            for k in range(len_buy, len_short):                queue = queue.append({'symbol':short_list[k],                                      'action':'SHORT'}, ignore_index=True)        print()        print(queue)    return results<jupyter_code>def backtest_lgb_v3(outcomes_new):
    
    retrain_interval = 50
    
    outcomes_new_dropna = outcomes_new.dropna()
    
    trade_date_list = sorted(list(set(outcomes_new.index.get_level_values('date'))))
    
    momentum_filter = 'close_scaled10'
    
    features_selected = info.features_lgb_v2  + [momentum_filter] #,'close_scaled10','close_scaled20','close_scaled30','close_scaled40']
    
    max_impact_pct = 0.001
    max_impact_volatility = 'price_chg_1_ema_std50'
    total_budget = context.capital_base * 1.2
    
    test_period = 250
    test_size = 1
    valid_size = 1
    training_size = 2000
    valid_test_gap = 4
    label = 'label_v2'

    #make a list of test dates according to test period, by back counting from the latest date with target
    test_dates = sorted(list(set(outcomes_new_dropna.index.get_level_values('date'))))[-test_period:]
    
    test_dates_converted = []
    for ts in test_dates:
        test_dates_converted.append(ts.to_pydatetime())
    
    results = pd.DataFrame()
    
    context.predictions = pd.DataFrame()
    
    day_count = 0
    
    for test_date in test_dates_converted:
        

        
        print()
        
        price_list = get_price_list(outcomes_new, test_date)
        
        price_list_open = get_price_list_open(outcomes_new, test_date)
        
        #update positions value according to open prices
        context.positions_value = positions_value(context.positions, price_list_open)
        
        if 'queue' not in globals() and 'queue' not in locals():
            queue = pd.DataFrame()
        
        context.max_positions = int((context.positions_value+context.cash)/(context.typical_amount * \
                                        (context.margin_ratio_long+context.margin_ratio_short)/2))
        context.max_daily_orders = int(context.max_positions/context.average_holding_period)
        context.max_daily_margin_limit = (context.positions_value+context.cash)/context.average_holding_period
        
        remaining_orders = context.max_daily_orders
        remaining_margin_limit = context.max_daily_margin_limit
        
        if len(queue)>0:
            while True:
                if len(queue)==0 or remaining_orders==0 or remaining_margin_limit<=0:
                    print ('Break with len(queue) = {}, remaining_orders = {} and remaining_margin_limit = {}'.format(len(queue),
                                                                                                                          remaining_orders,
                                                                                                                          remaining_margin_limit))
                    break
                order_list = list(queue[:min(remaining_orders, len(queue))]['symbol'])
                yesterday = trade_date_list[trade_date_list.index(test_date)-1]
                weights = inverse_volatility_weights(order_list, yesterday, outcomes_new)
                
                #place order
                symbol = queue[:1]['symbol'].values[0]
                order_type = queue[:1]['action'].values[0]
                order_id = ''
                num_shares = 1
                date_open = test_date
                traded_price = price_list_open[:,symbol].values[0]
                date_limit = trade_date_list[trade_date_list.index(date_open)+4]
                target_up = outcomes_new.loc[(date_open,symbol)]['target_upper_v2']
                target_down = outcomes_new.loc[(date_open,symbol)]['target_lower_v2']
                percentage = weights[symbol] if len(order_list)>1 else 1
                print()
                print("[{}] Symbol: {} Date open: {}, Date limit: {}, Traded price: {}.".format(order_type,
                                                                                                symbol,
                                                                                                date_open.strftime("%Y-%m-%d"),
                                                                                                date_limit.strftime("%Y-%m-%d"),
                                                                                                traded_price))
                order = Order(order_type, symbol, num_shares, date_open, traded_price, date_limit, target_up, target_down)
                status, margin_change = order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)
                if status == 'successful':
                    remaining_orders-=1
                    queue=queue.iloc[1:]
                    remaining_margin_limit-=margin_change
                    
                elif status == 'margin_overshoot':
                    break
                        
                elif status == 'min_charge_not_hitting':
                    #Cancel order
                    queue=queue.iloc[1:]

        #listen to market price of all stocks in positions
        #If hitting target price, then sell
        for i, row in context.positions.iterrows():
            if outcomes_new.loc[(test_date,row['symbol'])]['high']>row['target_up']:
                order_type = 'CLOSE'
                order_id = row['order_id']
                num_lots = 1
                num_shares = float('Nan')
                date_open = float('Nan')
                traded_price = row['target_up']
                date_limit = float('Nan')
                target_up = float('Nan')
                target_down = float('Nan')
                percentage = 0
                
                print()
                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)
                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)
                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)
                continue

            if outcomes_new.loc[(test_date,row['symbol'])]['low']<row['target_down']:
                order_type = 'CLOSE'
                order_id = row['order_id']
                num_lots = 1
                num_shares = float('Nan')
                date_open = float('Nan')
                traded_price = row['target_down']
                date_limit = float('Nan')
                target_up = float('Nan')
                target_down = float('Nan')
                percentage = 0
                
                print()
                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)
                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)
                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)
                continue
        
        #Near market close - check what stock in position, with date limit = today. Sell it at close price
        for i, row in context.positions.iterrows():
            #print('type of row[date_limit]', type(row['date_limit']))
            if test_date >= row['date_limit']:
                order_type = 'CLOSE'
                order_id = row['order_id']
                num_lots = 1
                num_shares = float('Nan')
                date_open = float('Nan')
                traded_price = price_list[:,row['symbol']].values[0]
                date_limit = float('Nan')
                target_up = float('Nan')
                target_down = float('Nan')
                percentage = 0
                
                print()
                print("Making CLOSE order.", test_date.strftime("%Y-%m-%d"), traded_price)
                order = Order(order_type, row['symbol'], num_shares, date_open, traded_price, date_limit, target_up, target_down)
                order_target_percent(order,  percentage, order_id, context, remaining_margin_limit)
        
        
        #After all the buy and sell (imagine the time now is after market close)
        #Update porfolio value
        context.positions_value = positions_value(context.positions, price_list)
        
        if 'symbol' in context.positions.columns:
            positions_list = list(context.positions['symbol'])
        else:
            positions_list = []
                                  
        result = {'positions': positions_list, #context.positions.to_dict('records'),
                'date': test_date,
                 'positions_value': context.positions_value,
                 'cash_value': context.cash,
                 'porfolio_value': context.positions_value+context.cash}
        
        results = results.append(result, ignore_index=True)
        
        #Update margin value due to the closing positions and STOCK VALUE CHANGE. USE CLOSE VALUE FOR THE DAY
        context.current_margin = positions_value_absolute(context.positions, price_list) * context.margin_ratio_long
        
        #CHECK MAINTENANCE MARGIN
        if context.positions_value+context.cash <= context.current_margin*0.8:
            print ('ELV lower than!!! MAINTENANCE MARGIN. GAME OVER LIAO.')
            return results

        #End of Day SMA calculation: ELV-Reg T Margin >= 0, cash>=-0.5*stock_value
        if context.positions_value+context.cash < context.positions_value * 0.5:
            print ('SMA smaller than zero! GAME OVER.')
            return results
        
        #HERE's the PREDICTION PART
        print('Test date: ', test_date)

        #get all trading dates from dataset of this symbol
        data_dates = sorted(list(set(outcomes_new_dropna.index.get_level_values(0))))
        data_converted_dates = []
        for ts in data_dates:
            data_converted_dates.append(ts.to_pydatetime())

        #Calculate training start date and valid start date by back counting
        start_date = data_converted_dates[data_converted_dates.index(test_date)-(valid_size + training_size + valid_test_gap + test_size)]
        start_date_valid = data_converted_dates[data_converted_dates.index(test_date)-(valid_size + valid_test_gap + test_size)]
        end_date_valid = data_converted_dates[data_converted_dates.index(test_date)-valid_test_gap-test_size]
        start_date_test = data_converted_dates[data_converted_dates.index(test_date)-test_size]
        end_date_test = test_date

        X_y_train, X_y_valid, X_y_test = train_valid_test_split(outcomes_new_dropna, start_date,
                                                                    start_date_valid, end_date_valid, start_date_test, end_date_test)

        #downsample the training set so each class has same number of samples
        X_y_train_resampled = downsample_3class(X_y_train, label, 42)

        #rename all -1 labels to 2 as LGB only takes 0,1,2...
        X_y_train_resampled.loc[X_y_train_resampled[label] == -1, label] = 2
        X_y_valid.loc[X_y_valid[label] == -1, label] = 2
        X_y_test.loc[X_y_test[label] == -1, label] = 2

        #split into features and target sets
        X_train, y_train = feature_target_split(X_y_train_resampled, features_selected, label)
        X_valid, y_valid = feature_target_split(X_y_valid, features_selected, label)
        X_test, y_test = feature_target_split(X_y_test, features_selected, label)

        train_data = lgb.Dataset(X_train, label=y_train)
        valid_data = lgb.Dataset(X_valid, label=y_valid)
        
        max_acc = float("-inf")
        boosting = 'gbdt'

        learning_rate = 0.006985117638031729
        min_data = 16
        num_leaves = 47
        max_depth = 7

        parameters = {
            'application': 'multiclass',
            'num_class': 3,
            #'is_unbalance': 'false',
            #'metric': 'auc',
            #'scale_pos_weight': 9,
            'boosting': boosting,
            'num_leaves': num_leaves,
            'feature_fraction': 0.95,
            'bagging_fraction': 0.2,
            'bagging_freq': 10,
            'learning_rate': learning_rate,
            'verbose': 0,
            'min_data_in_leaf': min_data,
            'max_depth': max_depth
        }
        
#         if day_count%50 == 0:
        model = lgb.train(parameters,
                               train_data,
                               valid_sets=valid_data,
                               num_boost_round=5000,
                                verbose_eval=False,
                               #feval=lgb_f1_score,
                               early_stopping_rounds=100)
            
        
        day_count += 1
        y_pred = model.predict(X_test, num_iteration=model.best_iteration)
        y_class_pred = [pred.argmax() for pred in y_pred]

        X_test['pred'] = y_class_pred
        buy_df = pd.DataFrame()
        short_df = pd.DataFrame()
        
        ##################################################
        ##################################################
        #search criteria for coming up with buy/short list
        search_criteria = 'momentum_25'
        ##################################################
        ##################################################
        
        for i, row in X_test.iterrows():
#             if row['pred'] == 1 and row[momentum_filter]>0:
#                 entry = {'symbol': i[1],
#                         'volatility': row['past_return_5_std50'],
#                         search_criteria: row[search_criteria]}
#                 buy_df=buy_df.append(entry, ignore_index=True)
                
            if row['pred'] == 2 and row[momentum_filter]<-0.5:
                entry = {'symbol': i[1],
                        'volatility': row['past_return_5_std50'],
                       search_criteria: row[search_criteria]}
                short_df=short_df.append(entry, ignore_index=True)
                
        if len(buy_df)!=0 or len(short_df)!=0:
            positive_pct = len(buy_df)/(len(buy_df)+len(short_df))
            negative_pct = len(short_df)/(len(buy_df)+len(short_df))
        else:
            positive_pct = 0
            negative_pct = 0
        
        print('Postive percentage: ', positive_pct)
        print('Negative percentage: ', negative_pct)
        print()
        
        if len(buy_df)>0:
            positive_list = list(buy_df['symbol'])
        else:
            positive_list = []
            
        if len(short_df)>0:
            negative_list = list(short_df['symbol'])
        else:
            negative_list = []
            
        predictions_entry = {'date':test_date,
                            'num_predictions_long':len(buy_df),
                            'num_predictions_short':len(short_df),
                            'predictions_long':positive_list,
                            'predictions_short':negative_list,
                            'percentage_long':positive_pct,
                            'percentage_short':negative_pct}
        
        context.predictions=context.predictions.append(predictions_entry, ignore_index=True)
        
        buy_drop_indexes = []
        short_drop_indexes = []
        
        for i, row in buy_df.iterrows():
            if len(context.positions)>0:
                if row['symbol'] in list(context.positions['symbol']):
                    buy_drop_indexes.append(i)

        for i, row in short_df.iterrows():
            if len(context.positions)>0:
                if row['symbol'] in list(context.positions['symbol']):
                    short_drop_indexes.append(i)
        
        buy_df=buy_df.drop(buy_drop_indexes)
        short_df=short_df.drop(short_drop_indexes)
        
        #search_criteria = 'volatility'
        
        buy_list = []
        short_list = []

        if len(buy_df)>0:
            buy_df = buy_df.sort_values(by=[search_criteria], ascending=False)
            buy_list = list(buy_df['symbol'])
        
        if len(short_df)>0:
            short_df = short_df.sort_values(by=[search_criteria], ascending=True)
            short_list = list(short_df['symbol'])
            
        print("--------------------------------")
        print('Buy_list symbols:')
        print(buy_list)
        print('Short_list symbols:')
        print(short_list)
        print("--------------------------------")
        
        len_buy = len(buy_list)
        len_short = len(short_list)
        
        queue = pd.DataFrame()
        
        #arrange buy and short list in a queue, alternately buy and short
        for symbol_to_buy, symbol_to_short in zip(buy_list, short_list):
            queue = queue.append({'symbol':symbol_to_buy,
                                  'action':'BUY'}, ignore_index=True)
            queue = queue.append({'symbol':symbol_to_short,
                                  'action':'SHORT'}, ignore_index=True)
        if len_buy>len_short:
            for k in range(len_short, len_buy):
                queue = queue.append({'symbol':buy_list[k],
                                      'action':'BUY'}, ignore_index=True)
        elif len_short>len_buy:
            for k in range(len_buy, len_short):
                queue = queue.append({'symbol':short_list[k],
                                      'action':'SHORT'}, ignore_index=True)
        print()
        print(queue)

    return results<jupyter_output><empty_output><jupyter_text>Consolidate the buy/short list to one long buy/Short alternated list with ALL PREDICTIONS Policy to deal with not enough money to get 1 lot. If targeted amount is more than 0.85 of min_charge of that stock, make it one lot. Else drop this orderQueue should have: 'BUY/SHORT', 'symbol', 'volatility', Next day start:update max_positionsupdate max_daily_ordersupdate max_daily_budgetupdate min_charges dictionaryremaining_daily_buget = max_daily_budgetremaining_orders = max_daily_ordersSet while loop to place order in the order of queue.* Break if counter meets remaining_orders==0 or len(queue)==0* Do inverse volatility on the first min(remaining_orders, len(queue)) items, USING YESTERDAYS' CLOSE, remaining_daily_buget* Place order.    * If successful, remaining_orders-=1, queue.pop(current_item), remaining_daily_budget-=amount, continue loop    * If unsuccessful        * If it is margin overshoot, break loop        * If it is not hitting min_charge            * If amount/min_charge > lot_threshold_fraction:                * Re-adjust the percentage to min_charge/remaining_daily_buget, place order again, remaining_orders-=1, queue.pop(current_item), remaining_daily_budget-=amount, continue loop             * Else:                 * queue.pop(current_item), continue loop  Volatility Parity Sizing in Core Trend Following model Number of shares to buy = Risk_Factor * Total_Porfolio_Value/StdSo that every position gives you same daily impact to the porfolio. 3 day settlement rule?<jupyter_code>context = Context()
initialize(context)
import time
#import random

start_time = time.time()

results = backtest_lgb_v3(outcomes_new)
#results = backtest_random(outcomes_new, context)
#results = backtest_rf(outcomes_new)

print("--- %s seconds ---" % (time.time() - start_time))
print(str(dt.timedelta(seconds=(time.time() - start_time))))
context.predictions
print('Time taken: {}'.format(str(dt.timedelta(seconds=(time.time() - start_time)))))
results
#v10 - position size depending on impact on porfolio
#v11 - changed impact percentage to 0.001, re-adjusted number of positions - fixed biased to number of short trade
#v12 - change back to weight based position sizing. Keeping adjusted number of positions.
#v13 - SHORT ONLY positions. Result: better then SHORT and LONG - BUT it's very volatile (risky)
#v14 - RANDOM, as a benchmark
#v15 - SHORT only and give momentum_100 threshold to be < -30
#v16 - both Long and short - give momentum_100 threshold of 10 for long and -10 for short
#v17 - constraint on momemtum_50 instead
#v18 - no momentum threshold, and 5 positions max for long and short respectively
#v19 - a few thresholds for Negative, no long buy. thresholds:
#                                   and row['momentum_5']<-60 \
#                                   and row['close_scaled50']<1.5\
#                                   and row['volume_scaled50']>1\
#                                   and row['bull_ratio']<0:  (NO pREDICITONs AT ALL!!!!!!)
#v20 - Buy long, no threshold, limit to 10 in buying list max
#v21 - Short only. no threshold, limit to 10 in short list max NEED REDO THE CURVE
#v22 - Buy and Short, no threshold, limit to 5 in short and long respectively RESULT NOT BAD
#v23 - Buy and Short, no threshold, limit to 10 in short and long respectively
#v24 - fixed wrong algo of creating short list. Before, was all or nothing.
#    - added simulation of margin. for each trade, check if ELV > 0.5 of current margin.
#    - If not, drop the order.  Iteration order is all buys first and then all shorts. For each end of day,
#    - check again ELV > 0.5 of all stock value (with udpated price) - NO GAME OVER
#v25 - change amount shared between 5 stocks to 0.6 of capital_base (300,000 of 500,000), aiming to diversify more.
#v26 - use momentum_100 as criteria to sort predictions
#v27 - use momentum_50 as criteria to sort predictions
#v28 - use momentum_25 as criteria to sort predictions
#v29 - change amount shared between 5 stocks to 0.4 of capital_base (200,000 of 500,000), aiming to diversify more.
#    - change back to use volatility as criteria
#v31 - same as v29, but use momentum_100 as sort criteria
#v32 - same as v29, but use momentum_50 as sort criteria
#v33 - same as v29, but use momentum_5 as sort criteria
#v34 - same as v29, but use momentum_25 as sort criteria SO FAR BEST PERFORMANCE
#v35 - same as v29, but use volatility as sort criteria
#v36 - Revamped the position assigning alogo. Buy/Short predictions will be consolidated to form an alternating buy/short list.
#      Place order according to new parameters: max_daily_orders, max_positions, max_daily_margin_limit, etc
#      search criteria: momentum_25
#v37 - same as v36, but use volatility as sort criteria
#v38 - same as v36, but use momentum_100 as sort criteria
#v39 - same as v36, but use momentum_25 as sort criteria and FORCE EQUAL NUMBER OF BUY/SHORT
#v40 - same as v36, but use momentum_50 as sort criteria, removed FORCE EQUAL Number

version = 'v41'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH +VE MOMENTUM_50 FOR LONG, and -VE FOR SHORT

version = 'v42'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH +VE MOMENTUM_5 FOR LONG, and -VE FOR SHORT

version = 'v43'
#same as v36, but use momentum_5 as sort criteria, and ONLY take STOCK WITH +VE MOMENTUM_5 FOR LONG, and -VE FOR SHORT

version = 'v44'
#same as v36, but use momentum_5 as sort criteria, and ONLY take STOCK WITH +VE MOMENTUM_50 FOR LONG, and -VE FOR SHORT

version = 'v45'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH +VE BULL_RATIO FOR LONG, and -VE FOR SHORT

version = 'v46'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH >1 close_vs_50d_high FOR LONG, and <1 FOR SHORT

version = 'v47'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH >0 close_scaled50 FOR LONG, and <0 FOR SHORT

version = 'v48'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH >0 close_scaled5 FOR LONG, and <0 FOR SHORT
#with 5 additional close_scaled features

version = 'v49'
#same as v36, but use momentum_50 as sort criteria, and ONLY take STOCK WITH >0 close_scaled10 FOR LONG, and <0 FOR SHORT
#with 5 additional close_scaled features

version = 'v50'
#same as v36, but use momentum_50 as sort criteria, and ONLY take close_scaled5<0 FOR SHORT. No filter for LONG
#with 5 additional close_scaled features

version = 'v51'
#same as v36, but use momentum_50 as sort criteria No filter

version = 'v52'
#same as v36, but use momentum_50 as sort criteria No filter (SAME AS ABOVE. TEST RANDOMNESS) Results: NO RANDOMNESS

version = 'v53'
#same as v36, but it's Random Forrest, use momentum_50 as sort criteria No filter

version = 'v54'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled5>0. No filter for Short

version = 'v55'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled10>0. No filter for Short

version = 'v56'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled20>0. No filter for Short

version = 'v57'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled30>0. No filter for Short

version = 'v58'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled10>0. No filter for Short.
#RETRAIN ONLY EVERY 50 days
#

version = 'v59'
#same as v36, LGB V3, use momentum_50 as sort criteria, filter for LONG: close_scaled10>0. No filter for Short.
#RETRAIN ONLY EVERY 50 days
#

version = 'v60'
#same as v36, LGB V3, use momentum_50 as sort criteria No filter for both.
#RETRAIN ONLY EVERY 50 days
#

version = 'v61'
#same as v36, LGB V3, use momentum_50 as sort criteria No filter for both.

version = 'v62'
#same as v36, LGB V3, use momentum_25 as sort criteria, filter for LONG: close_scaled10>0. No filter for Short

version = 'v63'
#same as v36, LGB V3, use momentum_25 as sort criteria, filter for LONG: close_scaled10>0. close_scaled10<0 for short

version = 'v64'
#same as v36, LGB V3, use momentum_25 as sort criteria LONG only no filter, no short

version = 'v65'
#same as v36, LGB V3, use momentum_25 as sort criteria LONG only w close_scaled10>0 filter, no short

version = 'v66'
#same as v36, LGB V3, use momentum_25 as sort criteria SHORT only no filter

version = 'v67'
#same as v36, LGB V3, use momentum_25 as sort criteria SHORT only  w close_scaled10<0, NO LONG

version = 'v68'
#same as v36, LGB V3, use momentum_25 as sort criteria SHORT only  w close_scaled10<-0.5, NO LONG

#####################################################################################################################
#v30 - Same as v29 except if order cancelled due to less than minimum num lot, next stock on queue will move up and take its place.
#    - Buy_list will not be limited by a quota and will include all predictions. But number of orders will be limited to 5

save_csv(results, path_pc, 'LGB_v2_result_long_short_'+version+'_250.csv')
len(context.predictions.loc[context.predictions['num_predictions_long']>=context.predictions['num_predictions_short']]), len(context.predictions)
outcomes_new.loc[('2019-05-15', '0386.HK')]['past_return_5_std50']
save_csv(context.predictions, path_pc, 'predictions_v12_250.csv')
outcomes_new.loc[outcomes_new['bull_ratio']<0]
len(outcomes_new)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09581718277463049, 'fraction_numerical': 0.016846275077735656, 'mean_word_length': 2.268708666898712, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 83, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '137821', 'license': 'MIT', 'n_tokens_mistral': 13648, 'n_tokens_neox': 12625, 'n_words': 3547, 'path': '.ipynb_checkpoints/Backtest with LightGBM-checkpoint.ipynb', 'repo_name': 'singyanchiu/LGBM-Stock-Prediction', 'size': '46954'}