<jupyter_start><jupyter_text>1. Loads data2. Estimates LRT 3. Censors all transition matrices4. Writes to disk Relies on the functions in `censoring.py`, which can also read data back in and re-normalize the censored transition matrices.   Setup<jupyter_code>%matplotlib inline
%load_ext autoreload
%autoreload 2
    
import time
import copy
from itertools import product
import numpy as np

import matplotlib.pyplot as plt
from cycler import cycler
color_list  = ['#1f77b4','#ff7f0e','#2ca02c','#d62728','#9467bd','#8c564b','#e377c2','#7f7f7f','#bcbd22','#17becf']
marker_list = ['o','x','d','v','^','<','>']
plt.rc('axes', prop_cycle=(cycler('color',color_list)))

from LRT import LRT
from LRT import moments
from LRT import figfuns
from LRT import PT
from LRT import PCTBIN

from LRT import censoring<jupyter_output><empty_output><jupyter_text>**Setup parameter struct:**<jupyter_code>par = LRT.ParStruct()
SAMPLE = '_p100'<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code># a. load
data = LRT.dataStruct()
data.logY = np.transpose(np.genfromtxt(f'data/logY{SAMPLE}.csv',delimiter=','))
T, par.N = data.logY.shape

# b. rescale
Y = np.exp(data.logY)
Y_lev = Y
Y = Y / np.mean(Y[0,:])
data.logY = np.log(Y)

# c. save simple data moments
mean_Y = np.ones(par.T)    
mean_Y_lev = np.ones(par.T)    
mean_logY = np.ones(par.T)
var_logY = np.ones(par.T)

for t in range(par.T):
    mean_Y[t] = np.mean(Y[t,:])
    mean_Y_lev[t] = np.mean(Y_lev[t,:])        
    mean_logY[t] = np.mean(data.logY[t,:])
    var_logY[t] = np.var(data.logY[t,:])
    
# d. upgrade simulation for large sample
par.simN = max([par.N,par.simN])<jupyter_output><empty_output><jupyter_text>Estimate   LRTPreferred specification is the depth 6. The amount of censoring (observations and cells in the transition matrices, averaged over all ages) from various depths is: * depth 5: 0.15% of observations; 15.07% of cells, * depth 6: 0.56% of observations; 16.56% of cells,* depth 7: 1.68% of observations; 14.53% of cells.<jupyter_code>models_LRT = []

for depth,name_short,name in zip([6],
                                 ['LRT_depth_6'],
                                 ['LRT (depth 6)']):

    # a. settings
    par.k = 5
    par.k_lead = 30    
    par.depth = depth

    # b. estimate
    model = LRT.estimate(par,data,name)
    model.name_short = name_short
    
    models_LRT.append(model)
prefmodel = models_LRT[0]<jupyter_output><empty_output><jupyter_text>Write the model to disk<jupyter_code>DOSAVE = True
min_obs_per_cell = 5
model = prefmodel
out_dir = f'censored_estimates'

censoring.write_model_to_disk(out_dir, model, min_obs_per_cell, DOCENSOR=True)<jupyter_output><empty_output><jupyter_text>Testing: reading back, simulating and plotting  Read back in<jupyter_code>m_load = censoring.read_censored_data(in_dir=out_dir, T=par.T, N=par.N)<jupyter_output><empty_output><jupyter_text>Simulate and computeTo verify that the censoring has not drastically impacted what simulations look like.<jupyter_code># simulate data
prefmodel.data = LRT.simulate(par, prefmodel, data)
m_load.data = censoring.simulate(m_load)
# compute moments 
for x in [data, m_load.data, prefmodel.data]: 
    moments.calc_all(par,x,printprogress=True)<jupyter_output><empty_output><jupyter_text>Plot<jupyter_code>prefmodel.name = 'LRT, exact'
m_load.name = 'LRT, censored'
prefmodel.marker = 'x'
prefmodel.color = color_list[1]
figfuns.age_profile(par, data, [prefmodel, m_load], 0, varname_str='logY', prefix='censoring')
figfuns.age_profile(par, data, [prefmodel, m_load], 5, 
                    varname_str='dlogY', prefix='censoring')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13322368421052633, 'fraction_numerical': 0.02138157894736842, 'mean_word_length': 4.866559485530547, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '356698', 'license': 'MIT', 'n_tokens_mistral': 1449, 'n_tokens_neox': 1376, 'n_words': 301, 'path': '2_7_censor_data_and_export.ipynb', 'repo_name': 'JeppeDruedahl/LRT', 'size': '3648'}