<jupyter_start><jupyter_text>Get the data froim the xarray objects and split between input and output for train and val sets.<jupyter_code>train_mean = train.mean(dim=('x','y')).dropna(dim='patch')
train_in = pd.DataFrame(train_mean['abi'].values)
train_out = train_mean['flash_counts'].values
train_out = np.where(train_out > 0, 1, 0)

val_mean = val.mean(dim=('x','y')).dropna(dim='patch')
val_in = pd.DataFrame(val_mean['abi'].values)
val_out = val_mean['flash_counts'].values
val_out = np.where(val_out > 0, 1, 0)
train_in.shape, train_out.shape, val_in.shape, val_out.shape
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error, r2_score, accuracy_score
from sklearn.decomposition import PCA
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.ensemble import RandomForestRegressor

scaler = StandardScaler()<jupyter_output><empty_output><jupyter_text>It is often helpful to have a persistance model - a very basic model - to compare results to. Here our persistance model will just always predict the majority class.  Now we have a metric to compare to to make sure we're making progress.<jupyter_code># this is just comaring our True output compared to always predicting Zero (the majority of cases)
print('Mean squared error: %.4f'
      % mean_squared_error(val_out, np.zeros(len(val_out)),squared=False))<jupyter_output>Mean squared error: 0.6669<jupyter_text>Perform Linear regression using the mean valus for each patch (for each band) as predictors. Much better than persistance model!<jupyter_code># Create linear regression object
mod = linear_model.LinearRegression()

# Train the model using the training sets
mod.fit(train_in, train_out)

# make predictions
preds = mod.predict(val_in)

print('Mean squared error: %.4f'
      % mean_squared_error(val_out, preds, squared=False))<jupyter_output>Mean squared error: 0.4057<jupyter_text>Perform Pricipal Component Analysis (PCA) to reduce dimensions of full data set. First well reshape the data into 2D data which the PCA function needs. Between the 4 bands and 32x32 patches we end up with 4096 features... Certainly enough to be reduced.<jupyter_code># reshape train and val sets for pca; display dimensions
t = np.array(train['abi'].values)
train_reshape = pd.DataFrame(t.reshape(t.shape[0], t.shape[1]*t.shape[2]*t.shape[3])).dropna()

v = np.array(val['abi'].values)
val_reshape = pd.DataFrame(v.reshape(v.shape[0], v.shape[1]*v.shape[2]*v.shape[3])).dropna()

train_reshape.shape, val_reshape.shape<jupyter_output><empty_output><jupyter_text>Perform Principal component analysis. We'll try to reduce down to 10 dimensions. We'll scale the data first.<jupyter_code>pca = PCA(n_components=10)
pca_train_in = scaler.fit_transform(pca.fit_transform(train_reshape))
pca_val_in = scaler.transform(pca.transform(val_reshape))

pca_train_in.shape, pca_val_in.shape<jupyter_output><empty_output><jupyter_text>Now we'll try a logistic regression model on the PCA input data. Slightly better reults, but not by much (as far as this metric goes anyway).<jupyter_code>logistic_mod = linear_model.LogisticRegression()
logistic_mod.fit(pca_train_in, train_out)

preds = logistic_mod.predict_proba(pca_val_in)[:,1]

print('Mean squared error: %.4f'
      % mean_squared_error(val_out, preds, squared=False))<jupyter_output>Mean squared error: 0.4031<jupyter_text>Try a Decision Tree Model... (Random Forest).  Better results!<jupyter_code>rf_mod = RandomForestRegressor()

# Train the model using the training sets
rf_mod.fit(pca_train_in, train_out)

preds = rf_mod.predict(pca_val_in)

print('Mean squared error: %.4f'
      % mean_squared_error(val_out, preds, squared=False))<jupyter_output>Mean squared error: 0.3520
{'fraction_non_alphanumeric': 0.10436178753010436, 'fraction_numerical': 0.014450093658014451, 'mean_word_length': 6.536290322580645, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '603391', 'license': 'MIT', 'n_tokens_mistral': 1328, 'n_tokens_neox': 1243, 'n_words': 339, 'path': 'notebooks/Example_NB_Day1.ipynb', 'repo_name': 'ShutingZhai/ai4ess-hackathon-2020', 'size': '3737'}