<jupyter_start><jupyter_text>Overnight returns [Overnight Returns and Firm-Specific Investor Sentiment](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2554010)> **Abtract**: We explore the possibility that overnight returns can serve as a measure of firm-specific investor sentiment by analyzing whether they exhibit characteristics expected of a sentiment measure. First, we document short-term persistence in overnight returns, consistent with existing evidence of short-term persistence in share demand of sentiment-influenced retail investors. Second, we find that short-term persistence is stronger for harder-to-value firms, consistent with evidence that sentiment plays a larger role when there is less objective data available for valuation. Third, we show that stocks with high (low) overnight returns underperform (outperform) over the longer-term, consistent with evidence of temporary sentiment-driven mispricing.  > **p 2, I**: The recent work of Berkman, Koch, Tuttle, and Zhang (2012) suggests that a stock’sovernight (close-to-open) return can serve as a measure of firm-level sentiment.> **p 3, I**: Specifically, Berkman et al. (2012) find that attention-generating events (high absolute returns orstrong net buying by retail investors) on one day lead to higher demand by individual investors,concentrated near the open of the next trading day...This creates temporary price pressure at theopen, resulting in elevated overnight returns that are reversed during the trading day.> **p 3, I**: We conduct three sets of analyses. **In the firstwe test for short-run persistence in overnight returns.** The basis for expecting this from ameasure of sentiment is the evidence in Barber et al. (2009) that the order imbalances of retailinvestors, who are the investors most likely to exhibit sentiment, persist for periods extendingover several weeks...In the third analysis weexamine whether stocks with high overnight returns underperform those with low overnightreturns over the long term.  Install packages<jupyter_code>import sys
!{sys.executable} -m pip install -r requirements.txt
import cvxpy as cvx
import numpy as np
import pandas as pd
import time
import os
import quiz_helper
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (14, 8)<jupyter_output><empty_output><jupyter_text>data bundle<jupyter_code>import os
import quiz_helper
from zipline.data import bundles
os.environ['ZIPLINE_ROOT'] = os.path.join(os.getcwd(), '..', '..','data','module_4_quizzes_eod')
ingest_func = bundles.csvdir.csvdir_equities(['daily'], quiz_helper.EOD_BUNDLE_NAME)
bundles.register(quiz_helper.EOD_BUNDLE_NAME, ingest_func)
print('Data Registered')<jupyter_output>Data Registered<jupyter_text>Build pipeline engine<jupyter_code>from zipline.pipeline import Pipeline
from zipline.pipeline.factors import AverageDollarVolume
from zipline.utils.calendars import get_calendar

universe = AverageDollarVolume(window_length=120).top(500) 
trading_calendar = get_calendar('NYSE') 
bundle_data = bundles.load(quiz_helper.EOD_BUNDLE_NAME)
engine = quiz_helper.build_pipeline_engine(bundle_data, trading_calendar)<jupyter_output><empty_output><jupyter_text>View Data¶With the pipeline engine built, let's get the stocks at the end of the period in the universe we're using. We'll use these tickers to generate the returns data for the our risk model.<jupyter_code>universe_end_date = pd.Timestamp('2016-01-05', tz='UTC')

universe_tickers = engine\
    .run_pipeline(
        Pipeline(screen=universe),
        universe_end_date,
        universe_end_date)\
    .index.get_level_values(1)\
    .values.tolist()
    
universe_tickers<jupyter_output><empty_output><jupyter_text>Get Returns data<jupyter_code>from zipline.data.data_portal import DataPortal

data_portal = DataPortal(
    bundle_data.asset_finder,
    trading_calendar=trading_calendar,
    first_trading_day=bundle_data.equity_daily_bar_reader.first_trading_day,
    equity_minute_reader=None,
    equity_daily_reader=bundle_data.equity_daily_bar_reader,
    adjustment_reader=bundle_data.adjustment_reader)<jupyter_output><empty_output><jupyter_text>Get pricing data helper function<jupyter_code>from quiz_helper import get_pricing<jupyter_output><empty_output><jupyter_text>get pricing data into a dataframe<jupyter_code>returns_df = \
    get_pricing(
        data_portal,
        trading_calendar,
        universe_tickers,
        universe_end_date - pd.DateOffset(years=5),
        universe_end_date)\
    .pct_change()[1:].fillna(0) #convert prices into returns

returns_df<jupyter_output><empty_output><jupyter_text>Sector data helper functionWe'll create an object for you, which defines a sector for each stock.  The sectors are represented by integers.  We inherit from the Classifier class.  [Documentation for Classifier](https://www.quantopian.com/posts/pipeline-classifiers-are-here), and the [source code for Classifier](https://github.com/quantopian/zipline/blob/master/zipline/pipeline/classifiers/classifier.py)<jupyter_code>from zipline.pipeline.classifiers import Classifier
from zipline.utils.numpy_utils import int64_dtype
class Sector(Classifier):
    dtype = int64_dtype
    window_length = 0
    inputs = ()
    missing_value = -1

    def __init__(self):
        self.data = np.load('../../data/project_4_sector/data.npy')

    def _compute(self, arrays, dates, assets, mask):
        return np.where(
            mask,
            self.data[assets],
            self.missing_value,
        )
sector = Sector()<jupyter_output><empty_output><jupyter_text>We'll use 2 years of data to calculate the factor **Note:** Going back 2 years falls on a day when the market is closed. Pipeline package doesn't handle start or end dates that don't fall on days when the market is open. To fix this, we went back 2 extra days to fall on the next day when the market is open.<jupyter_code>factor_start_date = universe_end_date - pd.DateOffset(years=2, days=2)
factor_start_date<jupyter_output><empty_output><jupyter_text>Walk through "Returns" classWe'll walk through how the `Returns` class works, because we'll create a new class that inherits from `Returns` in order to calculate a customized return. Returns inherits from CustomFactorThe zipline package has a class [zipline.pipeline.factors.Returns](https://www.zipline.io/appendix.html?highlight=returnszipline.pipeline.factors.Returns) which inherits from class [zipline.pipeline.CustomFactor](https://www.zipline.io/appendix.html?highlight=custom%20factorzipline.pipeline.CustomFactor).  The [source code for Returns is here](https://www.zipline.io/_modules/zipline/pipeline/factors/basic.htmlReturns), and the [source code for CustomFactor is here.](https://www.zipline.io/_modules/zipline/pipeline/factors/factor.htmlCustomFactor) **Please open the links to the documentation and source code and follow along with our notes about the code**  Inputs variableThe CustomFactor class takes the `inputs` as a parameter of the constructor for the class, otherwise it looks for a class-level variable named `inputs`.  `inputs` takes a list of BoundColumn instances.  These help us choose what kind of price-volume data to use as input.  The `Returns` class sets this to```inputs = [USEquityPricing.close]```  USEquityPricing classThe class [USEquityPricing](https://www.zipline.io/appendix.html?highlight=usequitypricingzipline.pipeline.data.USEquityPricing) has a couple BoundColumn instances that we can choose from.close = USEquityPricing.close  high = USEquityPricing.high  low = USEquityPricing.low  open = USEquityPricing.open  volume = USEquityPricing.volume    Quiz 1If we wish to calculate close to open returns, which columns from USEquityPricing do you think we'll want to put into the list and set as `inputs`?  Quiz 1 Answer`USEquityPricing.open` and `USEquityPricing.close`  window_length variableThe CustomFactor class takes `window_length` (an integer) as a constructor parameter, otherwise it looks for a class-level variable named `window_length`.  If we chose a `window_length = 2` then this means that we'll be passing two days' worth of data (two rows) into the `compute` function.  Quiz 2What window length would you choose if you were calculating daily close to open returns?  Assume we have daily data.  Answer 2window length of 2 to have 2 days of data.  Compute functionThe function definition of the `Returns` class includes the `compute` function```def compute(self, today, assets, out, close):        out[:] = (close[-1] - close[0]) / close[0]        ```* `today`: this is handled by parent classes; it has the datetime for the "today" row for the given subset of data.  We won't use it for this function implementation.* `assets`: this is handled by parent classes: it has the column header names for the "out" and "close".  We won't use it for this function implementation.* `out`: this points to a numpy array that will store the result of our compute.  It stores our "return" value of the `compute` function instead of explicitly returning a variable.* `*input`: a tuple of numpy arrays that contain input data that we'll use to compute a signal.  In the `Returns` definition of `compute`, the input is a single value `close`, but we can list more if we need additional columns of data to compute a return.If we set the `window_length=2`, then the `compute` function gets two rows worth of data from `close`.  The index 1 value is the most recent value, and the index 0 value is the earliest in time.  Recall that in Python, the -1 index is the same as getting the highest indexed value, so with a numpy array of just length two, -1 gives us the value at index 1.So the line of code is calculating the one-day return using the close price, and storing that into the `out` variable.$ Return = \frac{close_1 - close_0}{close_0} $   Quiz 3Given a numpy array for open prices called `open` and a numpy array for close prices called `close`, what code would you write to get the most recent open price?  Assume that you have 2 days of data.  Answer 3 open[1] or open[-1] (because python lets us do reverse indexing) are valid answers. Use the -1 index allows us to get the most recent price (the very last index of the numpy array) regardless of the window length, so you may prefer to use -1 to make your code easier to maintain or modify.  Close To Open Returns (Overnight Returns)The close-to-open return is the change in price between when the market closed on one day and when it opened on the next.  So it's$ CloseToOpen = \frac{open_1 - close_0}{close_0}$ We'll now create a class `CTO` that inherits from `Return`, and override the `compute` function.  Quiz 4Create a customized class `CloseToOpenReturns` that inherit from the Returns class.  Define the compute function to calculate overnight returns.<jupyter_code>from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.factors import Returns
class CloseToOpenReturns(Returns):
    """

    """
    
    # TODO: Set window_length (we're calculating daily returns)
    window_length = 2

    # TODO: set inputs
    inputs = [USEquityPricing.open, USEquityPricing.close]
    
    # The compute method is passed the current day, the assets list, a pre-allocated out vector, and the
    # factor's items in the list `inputs`
    def compute(self, today, assets, out, opens, closes):
        #TODO: calculate close-to-open return and save into out[:]
        out[:] = (opens[-1] - closes[0]) / closes[0]<jupyter_output><empty_output><jupyter_text>Trailing overnight returnsThe cumulative overnight returns over a week may be predictive of future returns; hence it's a kind of momentum signal.$ TrailingOvernightReturns = \sum_{1}^{Days}CloseToOpen_t$  Where $Days$ could be 5 if we are looking at a weekly window.So we want to take the `CloseToOpenReturns` as our input into another class, `TrailingOvernightReturns`, which also inherits from `Returns`.  maskNote that we're going to create another class that inherits from `Returns`.  Recall that `Returns` inherits from [CustomFactor](https://www.zipline.io/appendix.html?highlight=factorzipline.pipeline.CustomFactor), which has a `mask` parameter for its constructor.  The `mask` parameter takes in a `Filter` object, which determines which stock series get passed to the `compute` function.  Note that when we used `AverageDollarVolume` and stored its output in the variable `universe`, this `universe` variable is of type `Filter`.  Quiz 5If you wanted to create an object of type CloseToOpen, and also define the object so that it only computes returns on the set of stocks in universe that we selected earlier in this notebook, what code would you write?  Answer 5We could instantiate a CloseToOpen object with `CloseToOpenReturns(mask=universe)`, and this would only calculate close to open returns for the stocks defined in our `universe` variable.  numpy.nansumNumpy has a `nansum` function that treat NaN (not a number) as zeros.  Note that by default, if we give numpy.nansum a 2D numpy array, it will calculate a single sum across all rows and columns.  For our purposes, we want to compute a sum over 5 days (5 rows), and each column has daily close to open returns for a single stock.  It helps to think of the a matrix (2D numpy array) as a nested list of lists. This makes it easier to decide whether to set `axis=0` or `axis=1`.```tmp = [   [stock1day1, stock2day1 ]  [stock1day2, stock2day2 ]  ...]```If we look at the outermost list, each element is a list that represents one day's worth of data.  If we used `np.nansum(tmp,axis=0)`, this would sum across the days for each stock.  If we think of this as a 2D matrix, setting `axis=0` is like calculating a sum for each column.If we set `axis=0`, this applies `nansum` to the outermost list (axis 0), so that we end up with:```[  sum_of_stock_1, sum_of_stock_2]```Alternatively, if we set `axis=1`, this applies `nansum` to the lists nested inside the outermost list.  Each of these nested lists represent data for a single day, for all stocks, so that we get:```[  sum_of_day_1,  sum_of_day_2,]```  Example using numpy.nansum<jupyter_code>tmp = np.array([
                 [1, 2, 3],
                 [np.nan, np.nan, np.nan],
                 [1, 1, 1]
               ])

print(f"Sum across rows and columns: numpy.nansum(tmp) \n{np.nansum(tmp)}")
print(f"Sum for each column: numpy.nansum(tmp,axis=0) \n{np.nansum(tmp,axis=0)}")
print(f"Sum for each row: numpy.nansum(tmp,axis=1) \n{np.nansum(tmp,axis=1)}")<jupyter_output>Sum across rows and columns: numpy.nansum(tmp) 
9.0
Sum for each column: numpy.nansum(tmp,axis=0) 
[ 2.  3.  4.]
Sum for each row: numpy.nansum(tmp,axis=1) 
[ 6.  0.  3.]<jupyter_text>Quiz 6For our purposes, we want want a sum for each stock series.  Which axis do you think we should choose?  Answer 6We want to set axis = 0 so that we have a sum for each stock (each column).  Quiz 7Create a class TrailingOvernightReturns that inherits from Returns and takes the cumulative weekly sum of overnight returns.<jupyter_code>class TrailingOvernightReturns(Returns):
    """
    Sum of trailing close-to-open returns; we expect sentiment persistence at short horizons, so we
    look at the 5-day (ie., 1 week) window
    """
    # TODO: choose a window_length to calculate a weekly return
    window_length = 5
    
    # TODO: set inputs to a list containing the daily close to open returns
    # Filter the close to open returns by our stock universe
    inputs = [CloseToOpenReturns(mask=universe)]
    
    def compute(self, today, assets, out, close_to_open):
        #TODO: calculate the sum of close_to_open
        #choose the axis so that there is a sum for each stock (each column)
        #treat NaN as zeros
        out[:] = np.nansum(close_to_open, axis=0)<jupyter_output><empty_output><jupyter_text>Quiz 8Create a factor by instantiating the TrailingOvernightReturns class that you just defined.  Demean by sector, rank and covnert to a zscore.<jupyter_code># TODO: create an overnight_returns_factor variable
overnight_returns_factor = (
    TrailingOvernightReturns().
    demean(groupby=Sector()).
    rank().
    zscore()
)
# create a pipeline called p
p = Pipeline(screen=universe)
p.add(overnight_returns_factor, 'Overnight_Sentiment')<jupyter_output><empty_output><jupyter_text>Visualize pipeline<jupyter_code>p.show_graph(format='png')<jupyter_output><empty_output><jupyter_text>run pipeline and view the factor data<jupyter_code>df = engine.run_pipeline(p, factor_start_date, universe_end_date)
df.head()<jupyter_output><empty_output><jupyter_text>Visualize factor returnsThese are returns that a theoretical portfolio would have if its stock weights were determined by a single alpha factor's values.<jupyter_code>from quiz_helper import make_factor_plot
make_factor_plot(df, data_portal, trading_calendar, factor_start_date, universe_end_date);<jupyter_output>Formatting factor data for: Overnight_Sentiment
Dropped 0.4% entries from factor data: 0.4% in forward returns computation and 0.0% in binning phase (set max_loss=0 to see potentially suppressed Exceptions).
max_loss is 35.0%, not exceeded: OK!
Time to get arrange factor data: 3.53 secs
Time to generate long/short returns: 0.38 secs
{'fraction_non_alphanumeric': 0.07754723542366987, 'fraction_numerical': 0.00962095745913991, 'mean_word_length': 4.831361946603582, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 66, 'https://': 9, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '13127', 'license': 'MIT', 'n_tokens_mistral': 5096, 'n_tokens_neox': 4869, 'n_words': 2017, 'path': 'lesson-28/overnight_returns.ipynb', 'repo_name': 'rafaelmartinsbuck/ai-for-trading', 'size': '17254'}