<jupyter_start><jupyter_text>Show data_dict with column descriptions:<jupyter_code>print(f'Train shape = {df_train.shape} , test shape = {df_test.shape}')
df_test.date.describe()<jupyter_output><empty_output><jupyter_text>There are some columns with `output` in the name.. Those are not available at the test prediction, and thus not super useful yet. Although in TODO list I'd add a possibility to build the model using `output` predictors and compare on local CV the quality of predictions with the model that uses test columns only<jupyter_code>pd.set_option('display.max_rows',10)
data_dict<jupyter_output><empty_output><jupyter_text>Filter to features available both in train and test, and fill the NAs using padding left to right<jupyter_code>12 % 5
df_grouped = df_all_test.fillna(method = 'ffill')
df_grouped.shape
df_grouped['hour'] = df_grouped['date'].dt.hour

df_tsfresh = df_grouped.reset_index(drop=True).set_index('date')
df_tsfresh = df_tsfresh[(df_tsfresh.hour % 4) == 0].drop('hour',axis = 1)
df_tsfresh
from tsfresh.utilities.dataframe_functions import make_forecasting_frame
from tsfresh.utilities.dataframe_functions import impute
from tsfresh.feature_extraction import extract_features, MinimalFCParameters,EfficientFCParameters

feat_dict = { 'median': None,
 'mean': None,
'standard_deviation': None,
 'variance': None,
 'skewness': None,
 'kurtosis': None,
'quantile': [{'q': 0.1},
  {'q': 0.9}],
    'maximum': None,
 'minimum': None,
'linear_trend': [
  {'attr': 'rvalue'},
  {'attr': 'intercept'},
  {'attr': 'slope'}
]}
#df_features = extract_features(df_grouped[["primary_cleaner.state.floatbank8_a_air",'date','id']].head(1000), column_sort="date",column_id='id',n_jobs=4)
dfs = {}
for c in df_tsfresh.columns:
    print(f'Working on {c}...')
    df_shift, y = make_forecasting_frame(df_tsfresh[c], kind="price", max_timeshift=3*10, rolling_direction=1)
    X = extract_features(df_shift, column_id="id", column_sort="time", column_value="value",
                         impute_function=impute,show_warnings=False, default_fc_parameters=d,n_jobs=4)
    dfs[c] = X
cols = X.apply(np.any)
X.loc[:,cols].shape
dfs['primary_cleaner.input.copper_sulfate']
df_tsfresh_feats = pd.concat(dfs,keys = list(dfs.keys()))
df_tsfresh_feats.to_csv('../data/processed/tsfresh_feats_every4.csv')
a = df_grouped['date'].sample(10)
df_tsfresh_feats.head()<jupyter_output><empty_output><jupyter_text>Check correlation matrix of new features:<jupyter_code>from yellowbrick.features import Rank2D
fig,ax = plt.subplots(figsize = (16,16))
visualizer = Rank2D(ax = ax)
visualizer.fit(X,y=None)
visualizer.transform(X)
visualizer.poof()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13692480359147025, 'fraction_numerical': 0.011971567527123082, 'mean_word_length': 6.9347181008902075, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '208174', 'license': 'MIT', 'n_tokens_mistral': 992, 'n_tokens_neox': 948, 'n_words': 194, 'path': 'notebooks/build_features_test.ipynb', 'repo_name': 'AntonBiryukovUofC/unearthed-zinc', 'size': '2673'}