<jupyter_start><jupyter_text>Feature engineeringOur models can't reveal structure in our data when no structure exists.<jupyter_code>x1 = np.random.randn(100)
x2 = np.random.randn(100)
y = np.random.choice([True, False], size=100)

plt.figure(figsize=(8,8))
plt.plot(x1[y], x2[y], 'b.')
plt.plot(x1[~y], x2[~y], 'r.');
random_df = pd.DataFrame(np.vstack([x1, x2, y]).T, columns=['x1', 'x2', 'y'])
random_df.head()
random_df.corr()<jupyter_output><empty_output><jupyter_text>Beyond the choice of features to measure and how much data to collect (which we often can't control), there are many techniques for transforming our data set to amplify the most useful information for our machine learning models. Some of these techniques operate on the data directly while others are modifications to how we train our estimator. Such techniques are collectively called feature engineering.  Feature extractionMachine learning algorithm require observations to be vectors of features. Sometimes our raw data will not be a vector of features or a vector at all. For example, our data might be a collection of images. We could flatten the image into a vector, but this destroys important spatial relationships in the data. Our data could be a string of text like a news article. A string is linear, but doesn't have fixed length, and it's not immediately clear how to represent text as numbers. Our data could be several years of commodity price data, which is a vector of numbers, but it may be better represented in an aggregated form instead of a raw time-series.In cases like these we have the privilege of constructing features, giving us maximum control over preparation of the data (without actually deciding how raw data gets collected). Feature extraction will often be specific to the data set, but some things to think about include:- What counts as an observation in the raw data?- Are there spatial or temporal relationships within the data of a single observation?- If my data aren't numbers, what is the most meaningful way to transform the data into numbers?- If my data are numbers, are the raw values most meaningful? Differences between raw values? Other transformations or aggregations?Let's work through an example to illustrate some of these concepts.<jupyter_code>temps = pd.read_csv('./data/land_temps_by_city.csv', parse_dates=[0])
temps.head()<jupyter_output><empty_output><jupyter_text>We have temperature data for many cities around the world. If we were making predictions about the climate of these different cities, it may be more useful to summarize the temperature data in a quantity that characterizes the different climates. For example, some of these cities have 4 seasons while others do not. We might see this reflected in how the temperatures correlate with each other over the course of the year.<jupyter_code>from functools import reduce
def collect(x):
    return reduce(lambda y, z: y + [z] if isinstance(y, list) else [z], x)

def estimated_autocorrelation(x):
    n = len(x)
    variance = x.var()
    x = x-x.mean()
    r = np.correlate(x, x, mode = 'full')[-n:]
    result = r/(variance*(np.arange(n, 0, -1)))
    return result
plt.plot(estimated_autocorrelation(temps[temps['City']=='Berlin']['AverageTemperature'].values)[:12], label='Berlin')
plt.plot(estimated_autocorrelation(temps[temps['City']=='Jakarta']['AverageTemperature'].values)[:12], label='Jakarta')
plt.legend();<jupyter_output><empty_output><jupyter_text>Looking at these graphs, we can see that the temperature in Berlin six months in the future mirrors the temperature in Berlin today (correlation $\approx$ -1). However, in Jakarta, the average temperature six months in the future is basically unrelated to the temperature today (correlation $\approx$ 0). This indicates a different seasonality, associated with their different climates. We can extract the six-month autocorrelation value as a feature for each city's climate.<jupyter_code>ac = temps.groupby('City')['AverageTemperature'].apply(collect).apply(lambda x: np.array(x)).apply(estimated_autocorrelation).rename('autocorr')
ac_lat = pd.concat([ac.apply(lambda x: x[range(6, 1362, 12)].mean()), temps[['City', 'Latitude']].drop_duplicates().set_index('City')], axis=1)
ac_lat['Latitude'] = ac_lat['Latitude'].apply(lambda x: float(x[:-1]))
plt.plot(ac_lat['Latitude'], ac_lat['autocorr'], '.')
plt.xlabel('Latitude')
plt.ylabel('6-month temperature autocorrelation');

ax = plt.gca()
labels = {'Lima', 'Shenzhen', 'Delhi', 'Lima', 'Kingston', 'Cape Town', 'Jakarta', 'Nairobi', 'Rio De Janeiro', 'Quito', 'Port Moresby', 'Seoul', 'Moscow', 'Paris', 'Toronto'}
for city in ac_lat.index:
    if city in labels:
        plt.text(ac_lat.loc[city, 'Latitude'], ac_lat.loc[city, 'autocorr'] + .01, city)<jupyter_output><empty_output><jupyter_text>Feature transformationsSometimes our observations will be very unevenly distributed for a given feature. For example, income is roughly exponentially distributed in many populations. Sometimes the relationship between a feature and a regression target follows a clear non-linear trend. In cases like these it can be useful to transform the values of our features or our target to better highlight trends or to allow for use of models that might not otherwise be applicable.For example, we could fit a line to the relationship between latitude and 6-month autocorrelation in temperature, but it wouldn't be a great fit.<jupyter_code>from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression().fit(ac_lat['Latitude'].values.reshape(-1, 1), ac_lat['autocorr'].values.reshape(-1, 1))

ac_lat.sort_values('Latitude', inplace=True)

plt.plot(ac_lat['Latitude'], ac_lat['autocorr'], '.')
plt.plot(ac_lat['Latitude'], lin_reg.predict(ac_lat['Latitude'].values.reshape(-1, 1)))
plt.xlabel("Latitude")
plt.ylabel('6-month temperature autocorrelation');<jupyter_output><empty_output><jupyter_text>However, this data looks roughly like $y = e^{-x/\lambda} - 1$. If we substituted the variable $u = e^{-x/\lambda}$, we would have $y \propto u$, a linear relationship.<jupyter_code>plt.plot(np.exp(-ac_lat['Latitude']/13), ac_lat['autocorr'], '.')
plt.xlabel('Latitude')
plt.ylabel('6-month temperature autocorrelation');<jupyter_output><empty_output><jupyter_text>This looks somewhat more linear. We could fit a linear model to this transformed feature.<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error as mse
from sklearn.utils import shuffle

X_train, X_test, y_train, y_test = train_test_split(*shuffle(ac_lat['Latitude'].values.reshape(-1, 1), ac_lat['autocorr'].values.reshape(-1, 1)), test_size=0.1)

lin_reg = LinearRegression()

lin_reg.fit(np.exp(-X_train/13), y_train)

lin_pred = lin_reg.predict(np.exp(-X_test/13))<jupyter_output><empty_output><jupyter_text>Let's compare the result with a decision tree applied to the untransformed data.<jupyter_code>from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor(max_depth=3)
tree_reg.fit(X_train, y_train)
tree_pred = tree_reg.predict(X_test)
ac_lat.sort_values('Latitude', inplace=True)

plt.plot(ac_lat['Latitude'], ac_lat['autocorr'], '.', label='data')
plt.plot(ac_lat['Latitude'], lin_reg.predict(np.exp(-ac_lat['Latitude'].values.reshape(-1, 1) / 13)))
plt.plot(ac_lat['Latitude'], tree_reg.predict(ac_lat['Latitude'].values.reshape(-1, 1)))
plt.xlabel('Latitude')
plt.ylabel('6-month temperature autocorrelation');

print('Transformed linear regression: %0.2f' % mse(y_test, lin_pred))
print('Decision tree regression: %0.2f' % mse(y_test, tree_pred))<jupyter_output>Transformed linear regression: 0.01
Decision tree regression: 0.02<jupyter_text>Curse of dimensionalityIt is easy to see why adding features to a data set might be helpful.<jupyter_code>n = 1000
x1 = np.concatenate([np.random.randn(n // 2), np.random.randn(n // 2) + 2])
y = np.array([True if i < n // 2 else False for i in range(n)])

plt.figure(figsize=(8,8))
plt.hist(x1[y], alpha=.8)
plt.hist(x1[~y], alpha=.8);
x2 = np.concatenate([np.random.randn(n // 2), np.random.randn(n // 2) + 2])

plt.figure(figsize=(8,8))
plt.plot(x1[y], x2[y], 'b.')
plt.plot(x1[~y], x2[~y], 'r.');
np.vstack([x1, x2]).T
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(np.vstack([x1, x2]).T, y)

log_reg_1d = LogisticRegression(solver='lbfgs').fit(np.atleast_2d(X_train[:, 0]).T, y_train)

log_reg_2d = LogisticRegression(solver='lbfgs').fit(X_train, y_train)

print('Score on 1D data: %0.2f' % accuracy_score(y_test, log_reg_1d.predict(np.atleast_2d(X_test[:, 0]).T)))
print('Score on 2D data: %0.2f' % accuracy_score(y_test, log_reg_2d.predict(X_test)))
from matplotlib.colors import ListedColormap

xx1, xx2 = np.meshgrid(np.arange(-4, 6, .1), np.arange(-4, 6, .1))

Z = log_reg_2d.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
Z = Z.reshape(xx1.shape)

cut_1d = np.arange(-4, 6, .1)[log_reg_1d.predict_proba(np.atleast_2d(np.arange(-4,6,.1)).T)[:, 0] > .5][0]

colors = ('red', 'blue')
cmap = ListedColormap(colors)

plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
plt.plot(x1[y], x2[y], 'b.')
plt.plot(x1[~y], x2[~y], 'r.')
plt.axvline(cut_1d, color='k', linestyle='--');<jupyter_output><empty_output><jupyter_text>However, as we increase the dimensionality of our data by adding more features, the data gets more spread out. If the data is very high dimensional, it may be too spread out to identify any trends or clusters. This is know as the [curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality). As we increase the number of features, we will generally require more observations to ensure good sampling density throughout feature space.  Feature importance and selectionTo avoid the curse of dimensionality, we'll sometimes want to reduce the dimensionality of our data. We may also want to reduce the dimensionality of our data simply to compress it (in the case of very large data) or simply to visualize the data. Reducing dimensionality also helps make models more interpretable. When reducing the dimensionality of the data, we will want to get rid of irrelevant or redundant features.There are a number of ways to assess feature importance. Let's work again with the California housing data set.<jupyter_code>!wget http://dataincubator-wqu.s3.amazonaws.com/caldata/cal_housing.pkz -nc -P ~/scikit_learn_data/
from sklearn.datasets import fetch_california_housing

cali_data = fetch_california_housing()

cali_df = pd.DataFrame(cali_data.data, columns=cali_data.feature_names)
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import r2_score
import scipy.stats

X_train, X_test, y_train, y_test = train_test_split(*shuffle(cali_df, cali_data.target), test_size=0.1)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)

model = DecisionTreeRegressor()

rs = RandomizedSearchCV(model,
                  {'max_depth': scipy.stats.binom(9, .3, loc=1),
                  'min_samples_split': scipy.stats.binom(90, .5, loc=10)},
                   cv=5,
                   n_iter=200,
                   n_jobs=4,
                   scoring='neg_mean_squared_error')

rs.fit(X_train, y_train)
print(r2_score(y_test, rs.best_estimator_.predict(scaler.transform(X_test))))

list(zip(cali_data.feature_names, rs.best_estimator_.feature_importances_))
X_train, X_test, y_train, y_test = train_test_split(*shuffle(cali_df, cali_data.target), test_size=0.1)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)

lin_reg = LinearRegression().fit(X_train, y_train)

print(r2_score(y_test, lin_reg.predict(scaler.transform(X_test))))

list(zip(cali_data.feature_names, abs(lin_reg.coef_) / sum(abs(lin_reg.coef_))))<jupyter_output>0.6039322484914514<jupyter_text>The `feature_selection` submodule of Scikit-learn provides [some useful tools](http://scikit-learn.org/stable/modules/feature_selection.html) for discarding unimportant features.<jupyter_code>from sklearn.feature_selection import RFECV

feature_elimination = RFECV(estimator=model, cv=5, scoring='r2')
feature_elimination.fit(X_train, y_train)

rfecv_scores = feature_elimination.grid_scores_
plt.plot(range(1, len(rfecv_scores) + 1), rfecv_scores)
plt.xlabel('Number of features used')
plt.ylabel(r'Cross validation score ($R^2$)');<jupyter_output><empty_output><jupyter_text>RegularizationWhen we have too many features, our observations become sparse, making it hard to detect trends in the data. Our model begins to simply memorize the training set based on the many features. We can also say our model is fitting _noise_ instead of trend, because small random variations across many features results in observations being very separated.One way to limit overfitting due to high dimensionality is _regularization_. To regularize a model, we introduce a penalty in the cost function associated with the values of model parameters themselves. For example, we could regularize our linear regression model by changing the cost function.$$ C({\beta_i}) = \sum_j (y_j - X_{ij}\beta_i)^2 \LongrightarrowC({\beta_i}) = \sum_j (y_j - X_{ij}\beta_i)^2 + \alpha\sum_i\beta_i^2 $$The new cost function will tend to reduce the values of model parameters, limiting the effect of unimportant features. We could choose other penalties (e.g. $\alpha\sum_i\|\beta_i\|$) to change the effects of regularization.<jupyter_code>from sklearn.linear_model import Ridge, Lasso
X_train, X_test, y_train, y_test = train_test_split(*shuffle(cali_df, cali_data.target), test_size=0.1)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)

lin_reg = LinearRegression().fit(X_train, y_train)

alphas = np.logspace(-2, 2, 100)
ridge_coefs = []
ridge_scores = []
lasso_coefs = []
lasso_scores = []
for alpha in alphas:
    ridge_reg = Ridge(alpha=alpha).fit(X_train, y_train)
    lasso_reg = Lasso(alpha=alpha).fit(X_train, y_train)
    ridge_coefs.append(ridge_reg.coef_)
    ridge_scores.append(r2_score(y_test, ridge_reg.predict(scaler.transform(X_test))))
    lasso_coefs.append(lasso_reg.coef_)
    lasso_scores.append(r2_score(y_test, lasso_reg.predict(scaler.transform(X_test))))

lin_score = r2_score(y_test, lin_reg.predict(scaler.transform(X_test)))
print('Linear regression score: %0.2f' % lin_score)
print('Ridge regression score: %0.2f' % max(ridge_scores))
print('Lasso regression score: %0.2f' % max(lasso_scores))
plt.figure()
plt.gca().set_xscale('log')
plt.gca().set_ylim([.5, .625])
plt.plot(alphas, np.repeat(lin_score, len(alphas)), label='simple')
plt.plot(alphas, ridge_scores, label='ridge')
plt.plot(alphas, lasso_scores, label='lasso')
plt.xlabel(r'Regularization strength ($\alpha$)')
plt.ylabel(r'$R^2$ score')
plt.legend();
plt.figure(figsize=[12, 8])
plt.title('Regression coefficients')

plt.subplot(311)
plt.gca().set_xscale('log')
plt.ylabel('Simple regression coefficients')
for coef in lin_reg.coef_:
    plt.plot(alphas, np.repeat(coef, len(alphas)))

plt.subplot(312)
plt.gca().set_xscale('log')
plt.ylabel('Ridge regression coefficients')
plt.plot(alphas, ridge_coefs)

plt.subplot(313)
plt.gca().set_xscale('log')
plt.ylabel('Lasso regression coefficients')
plt.xlabel(r'Regularization strength ($\alpha$)')
plt.plot(alphas, lasso_coefs);
x = 15
print(alphas[x], lasso_scores[x])
list(zip(cali_data.feature_names, lasso_coefs[x]))<jupyter_output>0.040370172585965536 0.546595597614146<jupyter_text>Correlation between features and multicollinearitySometimes our features may be closely related to other features. We can easily calculate the correlation coefficient between pairs of features.<jupyter_code>corr = cali_df.corr()
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

cmap = sns.diverging_palette(220, 10, as_cmap=True)

sns.heatmap(corr, mask=mask, cmap=cmap, square=True, linewidths=.5, center=0, vmax=1);<jupyter_output><empty_output><jupyter_text>Correlation can be an indicator of feature redundancy. In particular if a feature can be approximated as a linear combination of other features$$ X_j \approx \sum_{i \ne j}\beta_{ij}X_i $$we say our features are multicollinear. Multicollinearity can introduce instability into our regression model parameters, making model interpretation difficult. Furthermore, features that are linear combinations of other features are redundant.  Principal component analysisSo far we have explored techniques for identifying unimportant or redundant features and ways to eliminate them from our data or model. When we eliminate features from our data, even if they are correlated with other features, we lose information.One way to mitigate the loss of information is to combine our $n$ original features into $m$ new features, with $m < n$. We can calculate what combinations preserve the most information using **principal component analysis** (PCA).Let's look back to an earlier example to understand how this works. We found earlier that adding a second feature to a data set helped us gain predictive power to separate two classes.<jupyter_code>from ipywidgets import interact

def plot_data(rotation=0):
    x1_r = x1 * np.cos(rotation) + x2 * np.sin(rotation)
    x2_r = -x1 * np.sin(rotation) + x2 * np.cos(rotation)
    plt.plot(x1_r[y], x2_r[y], 'b.')
    plt.plot(x1_r[~y], x2_r[~y], 'r.')

    w1, w2 = log_reg_2d.coef_[0]
    x = np.sort(x1)
    line = (-log_reg_2d.intercept_ - w1 * x) / w2
    x_r = x * np.cos(rotation) + line * np.sin(rotation)
    line_r = -x * np.sin(rotation) + line * np.cos(rotation)
    plt.plot(x_r, line_r)
    
interact(plot_data, rotation=(0, np.pi, .1));<jupyter_output><empty_output><jupyter_text>After rotating the data, we see that we could reduce the data from two dimensions to one without losing any important information. Principal component analysis projects the data into a lower dimensional space, choosing the projection by maximizing the variance in the projected data. Usually this is the optimal choice.<jupyter_code>from sklearn.decomposition import PCA

X_train, X_test, y_train, y_test = train_test_split(np.vstack([x1, x2]).T, y)

pca = PCA(n_components=1)
X_pca_train = pca.fit_transform(X_train)
X_pca_test = pca.transform(X_test)

log_reg_pca = LogisticRegression(solver='lbfgs').fit(X_pca_train, y_train)

print('Score on 2D data: %0.2f' % accuracy_score(y_test, log_reg_2d.predict(X_test)))
print('Score on 1D data (PCA): %0.2f' % accuracy_score(y_test, log_reg_pca.predict(X_pca_test)))<jupyter_output>Score on 2D data: 0.91
Score on 1D data (PCA): 0.91<jupyter_text>Ensemble modelsWe began the notebook by exploring feature extraction, which may involve applying complex algorithms to raw data to transform it into useful features for machine learning. An extension of this idea is to use machine learning models to construct features. This brings us to the idea of an _ensemble_ model, a model that uses the predictions of multiple estimators to make a final prediction.One example of an ensemble model is the _random forest_. A random forest model trains many decision trees on random pieces of the data and averages together their predictions. In a sense, the prediction of any individual tree becomes a feature in the final prediction.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV
X_train, X_test, y_train, y_test = train_test_split(*shuffle(cali_df, cali_data.target), test_size=0.1)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
model = RandomForestRegressor(n_estimators=50)

gs = GridSearchCV(model,
                  {'max_features': np.arange(.05, 1, .05)},
                  cv=5,
                  n_jobs=4,
                  scoring='neg_mean_squared_error')

gs.fit(X_train, y_train)

model = gs.best_estimator_

print(gs.best_params_)
print(r2_score(y_test, model.predict(scaler.transform(X_test))))
gs = GridSearchCV(model,
                  {'min_samples_leaf': np.arange(1, 50, 5)},
                  cv=5,
                  n_jobs=4,
                  scoring='neg_mean_squared_error')

gs.fit(X_train, y_train)

model = gs.best_estimator_

print(gs.best_params_)
print(r2_score(y_test, model.predict(scaler.transform(X_test))))
gs = GridSearchCV(model,
                  {'n_estimators': np.arange(100, 301, 100)},
                  cv=5,
                  n_jobs=4,
                  scoring='neg_mean_squared_error')

gs.fit(X_train, y_train)

model = gs.best_estimator_

print(gs.best_params_)
print(r2_score(y_test, model.predict(scaler.transform(X_test))))<jupyter_output>exception calling callback for <Future at 0x7f66f1d69c88 state=finished raised TerminatedWorkerError>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "/opt/conda/lib/python3.7/site-packages/joblib/parallel.py", line 340, in __call__
    self.parallel.dispatch_next()
  File "/opt/conda/lib/python3.7/site-packages/joblib/parallel.py", line 769, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/opt/conda/lib/python3.7/site-packages/joblib/parallel.py", line 835, in dispatch_one_batch
    self._dispatch(tasks)
  File "/opt/conda/lib/python3.7/site-packages/joblib/parallel.py", line 754, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/opt/conda/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 551, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/opt/conda/lib/[...]<jupyter_text>Custom ensemblesWe can also create our own ensembles.This will involve some new tools, _pipelines_ and _feature unions_. Pipelines allow us to bundle a series of transformers and estimators. Feature unions all us to bundle parallel transformers.Remember that our ensemble is not meant to make predictions per se, but rather perform feature extraction, similar to a transformer. Therefore we need the estimators in our ensemble to implement a `transform` method instead of a `predict` method. We can make this modification using a simple custom class.<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

class EstTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, estimator):
        self.estimator = estimator
    
    def fit(self, X, y):
        self.estimator.fit(X, y)
        return self

    def transform(self, X):
        return np.atleast_2d(self.estimator.predict(X)).T<jupyter_output><empty_output><jupyter_text>Now we can build a feature union that will use a decision tree regressor and a K-neighbors regressor in parallel to construct our ensemble features. Our pipeline will scale our raw features, pass them to the ensemble, and then make a final prediction using a simple linear regression to combine the ensemble predictions.<jupyter_code>from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(*shuffle(cali_df, cali_data.target), test_size=0.1)

knn = Pipeline([
    ('scaler', StandardScaler()),
    ('knn', KNeighborsRegressor())
])

gs = GridSearchCV(knn,
                  {'knn__n_neighbors': range(5, 26, 5)},
                  cv=5,
                  n_jobs=4,
                  scoring='neg_mean_squared_error')

gs.fit(X_train, y_train)
print(gs.best_params_)
print(r2_score(y_test, gs.best_estimator_.predict(X_test)))
from sklearn.pipeline import FeatureUnion

ensemble = FeatureUnion([
    ('rf', EstTransformer(DecisionTreeRegressor(max_depth=5, min_samples_split=45))),
    ('knn', EstTransformer(KNeighborsRegressor(n_neighbors=10)))
])

model = Pipeline([
    ('scaler', StandardScaler()),
    ('ensemble', ensemble),
    ('combine', LinearRegression(fit_intercept=False))
    ])

model.fit(X_train, y_train)

print(r2_score(y_test, model.predict(X_test)))<jupyter_output>0.7217738548688355
{'fraction_non_alphanumeric': 0.11171537695117902, 'fraction_numerical': 0.017643639986715375, 'mean_word_length': 5.371065855593758, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '653877', 'license': 'MIT', 'n_tokens_mistral': 8170, 'n_tokens_neox': 7710, 'n_words': 2286, 'path': '7_ML_FeatureEngineering.ipynb', 'repo_name': 'jhonsonlee/basics-of-machine-learning', 'size': '24088'}