<jupyter_start><jupyter_text>Python implementation ofShah, Anish, Easy Way to Merge Return Forecasts across Securities and Horizons (September 24, 2019). Available at SSRN: https://ssrn.com/abstract=3459184 or http://dx.doi.org/10.2139/ssrn.3459184<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>1. Create dummy data<jupyter_code>def dummy_forecasts(m, n):
  # function creates dummy forecast data to test forecast combination
  # m = # of forecasts
  # n = # of securities
  # returns:
  # start = (m x 1) start periods for forecasts
  # end = (m x 1) end periods for forecasts
  # P = (m x n) linear combinations forecasted
  # y = (m x 1) return forecasts
  # H = (m x m) forecast noise covariance
  # mu = (n x 1) prior mean of 1 period returns
  # C = (n x n) prior covariance of 1 period returns

  start = np.random.randint(low=0,high=10,size=m)
  end = start + np.random.randint(low=1,high=5,size=m)

  s = 0.1
  y = s * np.random.random(size=[m,1])
  I = np.identity(n)
  k = int(np.floor(1.5*m))
  r = np.random.randint(low=0, high=n, size=k)
  P = I[r[:m],:]  # forecast random individual securities
  P[-(k-m):,:] = P[-(k-m):,:] - I[r[m:],:]  # make some spreads
  y[-(k-m):] -= s * 0.5 
  v = np.random.random(n)
  v = np.around(v / v.sum(), 2)
  v[np.argmax(v)] -= v.sum() - 1
  P[0,:] = v # make first entry a portfolio

  Q = np.around(np.random.random(size=[m,m]), 2)
  H = 10. * Q.dot(Q.T)

  Q = np.around(np.random.random(size=[n,n]), 2)
  C = Q.dot(Q.T)
  mu = np.around(0.01*(np.random.random(size=[n,1]) - 0.5), 3)
  return start, end, P, y, H, mu, C
# m = # of forecasts
# n = # of securities
#
# start = (m x 1) start periods for forecasts
# end = (m x 1) end periods for forecasts
# P = (m x n) linear combinations forecasted
# y = (m x 1) return forecasts
# H = (m x m) forecast noise covariance
# mu = (n x 1) prior mean of 1 period returns
# C = (n x n) prior covariance of 1 period returns

m = 5  # number of forecasts
n = 20 # number of securities
start, end, P, y, H, mu, C = dummy_forecasts(m, n)<jupyter_output><empty_output><jupyter_text>2. Segment objects being forecasted into time segments. Then calculate posterior mean and covariance given the forecasts<jupyter_code># model is y = P x + eps  where eps ~ N(0, H)
# x occurs over the intervals start -> end  and prior x over 1 period ~ N(mu, C)
# want mean and cov of x over different horizons | y
def calculate_posterior_of_segments(start, end, P, H, mu, C, more_horizons=[0.]):

  # more_horizons = (list) of points to consider in addition to start and end
  #               = [0., 5., 21.]  # e.g. to be able to get 1 week and 1 month return forecasts
  m, n = P.shape # num of forecasts x num of securities

  # points in ascending order where time needs to be segmented
  # Tpts = np.unique((start,end))
  Tpts = np.unique([t for x in [start, end, more_horizons] for t in x])

  # put forecast start and end in terms of time markers
  startpt = np.searchsorted(Tpts, start)
  endpt = np.searchsorted(Tpts, end)
  assert np.alltrue(Tpts[startpt] == start)
  assert np.alltrue(Tpts[endpt] == end)

  # break quantities being forecasted into time segments
  # e.g. r(0->T) = r(0->T1) + r(T1->T2) + ... + r(Tk-1->T)
  nseg = len(Tpts) - 1
  nsegvars = n*nseg
  Z = np.zeros((m, nsegvars)) # matrix that will hold forecasts in terms of segments
  nu = np.zeros((nsegvars,1)) # vector that will hold mean for each segment variable
  Omega = np.zeros((nsegvars,nsegvars)) # matrix that will hold cov of segment variables
  for i in range(nseg):
    l = Tpts[i+1] - Tpts[i]   # number of time periods in segment
    sidx = i*n       # start index of variables in time segment
    eidx = sidx + n  # end index of variables in time segment
    nu[sidx:eidx,:] = l * mu            # mean over segment
    Omega[sidx:eidx,sidx:eidx] = l * C  # variance over segment
    inseg = (startpt <= i) & (endpt >= i+1)  # True for forecasts that contain segment
    Z[inseg, sidx:eidx] = P[inseg,:]    # put coefficients on segment vars involved in forecasts

  # now have everything to calculate posterior distribution
  ZOmega = Z.dot(Omega)
  F = ZOmega.dot(Z.T) + H
  # B = Omega Z' inv(F), and F and Omega are symmetric
  # B = (ZOmega.T).dot(np.linalg.inv(F))
  B = np.linalg.solve(F, ZOmega).T  # computationally better this way

  # segment variables given forecasts have
  # mean = a_mean + B y   = nu + B (y - Z nu)   where y is the vector of forecasts
  # & cov = Sigma
  Sigma = Omega - B.dot(ZOmega)
  a_mean = nu - B.dot(Z.dot(nu))
  return Tpts, a_mean, B, Sigma
Tpts, a_mean, B, Sigma = calculate_posterior_of_segments(start, end, P, H, mu, C)<jupyter_output><empty_output><jupyter_text>3. Now can calculate the mean and cov of any linear combinations of segment variables given the forecasts<jupyter_code>m, n = P.shape
nTps = len(Tpts) 
nsegvars = B.shape[0]

print(nTps, "Tpts -", Tpts, "- so", nTps-1, "segments")
print(n, "securities,", nsegvars, "security segments")
print(m, "forecasts")
print("a_mean.shape:", a_mean.shape)
print("B.shape:", B.shape)
print("Sigma.shape", Sigma.shape)

# segment variables given forecasts have
# mean = a_mean + B y  and  cov = Sigma    where y is the vector of forecasts<jupyter_output>8 Tpts - [ 0.  4.  5.  6.  7.  8.  9. 10.] - so 7 segments
20 securities, 140 security segments
5 forecasts
a_mean.shape: (140, 1)
B.shape: (140, 5)
Sigma.shape (140, 140)<jupyter_text>A. Example: all the securities over the interval between the 2nd and 4th Tpt<jupyter_code>A = np.zeros((n, nsegvars))
A[:,n:2*n] = np.identity(n)
A[:,2*n:3*n] = np.identity(n)
# given the forecasts, these segment variable combinations
# have mean m0 + M y and cov A Sigma A'
m0 = A.dot(a_mean)
M = A.dot(B)   # tells how much each forecast contributed
pmean = m0 + M.dot(y)
pcov = A.dot(Sigma).dot(A.T)
print(pmean.shape)
print(pcov.shape)<jupyter_output>(20, 1)
(20, 20)<jupyter_text>B. Example: the first security over each separate segment<jupyter_code>nsegs = nTps-1
A = np.zeros((nsegs, nsegvars))
secnum = 0 # first security
for i in range(nsegs):
  A[i, i*n + secnum] = 1.   # each row in A is a different interval of the same security
# given the forecasts, these segment variable combinations
# have mean m0 + M y and cov A Sigma A'
m0 = A.dot(a_mean)
M = A.dot(B)   # tells how much each forecast contributed
pmean = m0 + M.dot(y)
pcov = A.dot(Sigma).dot(A.T)
print(pmean.shape)
print(pcov.shape)<jupyter_output>(7, 1)
(7, 7)
{'fraction_non_alphanumeric': 0.13408598824621096, 'fraction_numerical': 0.022888957624497372, 'mean_word_length': 3.5995732574679944, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '645802', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2547, 'n_tokens_neox': 2371, 'n_words': 808, 'path': 'merge_forecasts.ipynb', 'repo_name': 'AnishRSIGM/CombineForecasts', 'size': '6466'}