<jupyter_start><jupyter_text>Advanced Dimensionality Reduction: SVD Mis<NAME>  Executive SummarySubsurface spatial data is often very large and difficult to process and utilize in machine learning workflows. Moreover, there tends to be high degrees of correlation and structure in 2D subsurface data, which makes it a perfect candidate for latent structure analysis and modeling. Here, we will utilize a 2D multivariate subsurface dataset and demonstrate the usage of **Singular Value Decomposition** to encode the data into the latent space and then reconstruct using a fraction of the original size. Singular Value Decomposition works by decomposing the data matrix into three components: left-singular vectors, singular values, and right-singular vectors. We can then project onto the first few singular values/vectors to obtain a reduced-dimensionality representation of the data, and use this new feature space to approximately reconstruct our images.We learn that using SVD, different latent dimensions will result in different degrees of lossy reconstruction, but that this method is efficient, economic, and simple enough to treat our large data into a reduced dimensionality form. We recommend to use this autoencoder structure whenever dealing with image or volume problems in order to reduce redundancy and increase efficiency of our machine learning workflows.  Table of Contents1. Import Packages 2. Declare Functions 3. Load & Preprocess Data        a) MNIST Data         b) Subsurface Data4. Dimensionality Reduction: Singular Value Decomposition (SVD)  ***  1. Import Packages We start by importing our most basic packages:<jupyter_code>%matplotlib inline
import numpy as np                     #arrays and matrix math
import pandas as pd                    #DataFrames management and indexing
import matplotlib.pyplot as plt        #plotting and visualization
import matplotlib.gridspec as gridspec #enhanced subplot referencing
import tensorflow as tf                #deep learning functionality and MNIST data<jupyter_output><empty_output><jupyter_text>Import other important packages for preprocessing, metrics, etc., and project-specific packages and functions<jupyter_code># Feature Engineering/Preprocessing
from sklearn.preprocessing import MinMaxScaler                #Normalize variables to be in [0,1]
from scipy.interpolate import Rbf as Rbf_interpolation        #Inteprolate 2D map from sparse data

# Goodness-of-Fit Metrics
from sklearn.metrics import mean_squared_error                #Mean squared error (MSE)
from skimage.metrics import structural_similarity as SSIM     #Structural Similarity Index (SSIM)

# Project-specific
from numpy.linalg import svd as SVD                           #Singular Value Decomposition<jupyter_output><empty_output><jupyter_text>2. Delcare FunctionsThe following functions will be used in the workflow.<jupyter_code># Plot function for sample images
def plot_sample_matrix(samples, my_cmap):
    num_samples, x_dim, y_dim, _ = samples.shape
    axes = (np.round(np.sqrt(num_samples))).astype(int)
    fig  = plt.figure(figsize=(axes, axes))
    gs   = gridspec.GridSpec(axes, axes)
    gs.update(wspace=0.05, hspace=0.05)
    for i, sample in enumerate(samples):
        ax = plt.subplot(gs[i])
        plt.axis('off'); ax.set_aspect('equal')
        plt.imshow(sample, cmap=my_cmap)<jupyter_output><empty_output><jupyter_text>This variable will help us when inserting text boxes into our matplotlib plots<jupyter_code># Define arguments for text box in PLT.TEXT()
my_box = dict(boxstyle='round', facecolor='wheat', alpha=0.5)<jupyter_output><empty_output><jupyter_text>This next function is optional, and will simply be a check that your current version of tensorflow on your Python kernel is running on a GPU and if tensorflow is built with CUDA.<jupyter_code># Check tensorflow GPU settings
print('Tensorflow built with CUDA? ',  tf.test.is_built_with_cuda())
tf.config.list_physical_devices()<jupyter_output>Tensorflow built with CUDA?  True<jupyter_text>3. Load & Preprocess DataWe will deal with two different datasets, both of which need preprocessing.(1) MNIST dataset: handwritten digits as $28x28$ images from *tensorflow* (2) Subsurface multivariate data: 2D spatial (sparse) data  3. a) MNIST DataThis is a set of $60,000$ images of handwritten digits $0$-$9$. We load it directly from *tensorflow* datasets ([link](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist)), and will preprocess to center and flatten as needed for our techniques.<jupyter_code># Load the Dataset and split into train/test
(x_train_all, y_train_all), (x_test_all, y_test_all) = tf.keras.datasets.mnist.load_data()

# Choose to work with ALL or only a few (N) MNIST images (full size is 60,000)
#N = len(x_train_all)
N = 5000
x_train, x_test = x_train_all[:N], x_test_all[:N]
y_train, y_test = y_train_all[:N], y_test_all[:N]

# Normalize the Images
x_train = np.expand_dims(x_train/255.0, axis=-1)
x_test  = np.expand_dims(x_test/255.0, axis=-1)

# Define the labels
class_names = ['zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine']

# Print the shapes of the training and testing sets + check that training images are normalized
print('MNIST dataset properties:')
print('Train || Shapes: X={}, Y={} | min={}, max={}'.format(x_train.shape, y_train.shape, x_train.min(), x_train.max()))
print('Test  || Shapes: X={}, Y={} | min={}, max={}'.format(x_test.shape, y_test.shape, x_test.min(), x_test.max()))
# Flatten and Center the images
print('Flattened and Center Images:')

# Flatten the images into NxM array
x_train_f = np.transpose(np.reshape(x_train, [x_train.shape[0], -1]))
x_test_f  = np.transpose(np.reshape(x_test, [x_test.shape[0], -1]))

# Center the Flattened images
x_train_f_c = x_train_f - np.expand_dims(np.mean(x_train_f, axis=1), axis=1)
x_test_f_c  = x_test_f - np.expand_dims(np.mean(x_test_f, axis=1), axis=1)

print('Train || Shapes: X={}, Y={} | min={:.3f}, max={:.3f}'.format(x_train_f_c.shape, y_train.shape, x_train_f_c.min(), x_train_f_c.max()))
print('Test  || Shapes: X={}, Y={} | min={:.3f}, max={:.3f}'.format(x_test_f_c.shape, y_test.shape, x_test_f_c.min(), x_test_f_c.max()))<jupyter_output>Flattened and Center Images:
Train || Shapes: X=(784, 5000), Y=(5000,) | min=-0.549, max=1.000
Test  || Shapes: X=(784, 5000), Y=(5000,) | min=-0.531, max=1.000<jupyter_text>For improved visualization, we will define a new colormap that uses the 10 individual digits ($0$-$9$) and implement the '*jet*' colormap.<jupyter_code># Define a colormap for the 10-class classification system
import matplotlib.cm as cm
from matplotlib.colors import Normalize
my_cmap = cm.get_cmap('jet')
my_norm = Normalize(vmin=0, vmax=9)
cs      = my_cmap(my_norm(y_train))<jupyter_output><empty_output><jupyter_text>Next, we will count the number of items that is in each of the 10 digit categories, and also visualize the first few samples from the training and testing dataset.<jupyter_code># Count the number of occurrences for each digit within the training/testing datasets
digit_count = {}
for i in np.arange(len(class_names)):
    digit_count[i] = x_train[y_train==i].shape[0]
digit_count = pd.DataFrame(list(digit_count.values()), columns=['Count']).T
print('Count per Digit:')
digit_count.head()
# Visualize a few Train/Test samples from mnist
fig = plt.figure(figsize=(10, 3), constrained_layout=False)
fig.suptitle('Train samples'+60*' '+'Test samples')
outer_grid = fig.add_gridspec(1, 2, wspace=0.1, hspace=0)

left_grid = outer_grid[0, 0].subgridspec(5, 10, wspace=0, hspace=0)
axs = left_grid.subplots() 
for (c,d), ax in np.ndenumerate(axs):
    ax.imshow(x_train[y_train==d][c]); ax.set(xticks=[], yticks=[])
    
right_grid = outer_grid[0, 1].subgridspec(5, 10, wspace=0, hspace=0)
axs = right_grid.subplots() 
for (c,d), ax in np.ndenumerate(axs):
    ax.imshow(x_test[y_test==d][c]); ax.set(xticks=[], yticks=[])
    
plt.show();<jupyter_output><empty_output><jupyter_text>3. b) Subsurface DataThe following workflow applies the .csv file 'spatial_nonlinear_MV_facies_v1.csv', a synthetic dataset calculated with geostatistical cosimulation by Dr. <NAME>, The University of Texas at Austin. The dataset is publically available [here](https://github.com/GeostatsGuy/GeoDataSets)  From this site, other datasets can also be used for this workflow including but not limited to: {'spatial_nonlinear_MV_facies_v5.csv', 'sample_data_MV_biased.csv', 'PGE383_Dataset_13_Wells.csv', '12_sample_data.csv'}.We will work with the following features:* **X** and **Y** - the sptial coordinates (in meters) for the subsurface data* **Porosity** - fraction of rock void in units of percentage* **Permeability** - ability of a fluid to flow through the rock in milliDarcy* **Acoustic Impedence** - product of sonic velocity and rock density (in $kg/m^2s*10^3$)* **Facies** - binary indicator of sand or shale facies<jupyter_code># Select a subsurface Dataset for image reconstruction
df = pd.read_csv("https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/spatial_nonlinear_MV_facies_v1.csv")
df.head() #visualize first few rows of the DataFrame<jupyter_output><empty_output><jupyter_text>We perform normalization of the features by applying Min-Max scaling of the features such that:$$ x^* = \frac{x-min(x)}{max(x)-min(x)} $$where $min(x)$ and $max(x)$ are the minimum and maximum values for each of the features in the dataset.This is done by the *scikitlearn* built-in function``` pythonmin_max_scaler = sklearn.preprocessing.MinMaxScaler()scaled_array   = min_max_scaler.fit_transform(float_array)```<jupyter_code>scaler = MinMaxScaler()                           #instantiate the normalization function
df_s = pd.DataFrame(scaler.fit_transform(df), columns=df.columns)  #apply min-max scaling
df_s.describe().T                           #show summary statistics of the new DataFrame<jupyter_output><empty_output><jupyter_text>For simplicity, we specifically name our subsurface features/properties.We also specifically name the *matplotlib* colormaps that we want to use for each of the feautures/properties.<jupyter_code>features = ['Porosity','Perm','AI','Facies']       # names of our features
my_maps  = ['magma', 'jet', 'seismic', 'viridis']  # names of the corresponding colormaps<jupyter_output><empty_output><jupyter_text>However, this is a **sparse** dataset of the subsurface, with 457 wells in approximately $1 km^2$ area of the subsurface. Therefore, we must interpolate the spatial properties so that we obtain a full image of the subsurface properties. The 2D interpolation is done through *scipy*'s RBF interpolation function. This generates radial basis function inteprolation from $(N,D)$ arrays to an $(M,D)$ domain.We will interpolate the subsurface 2D data into $(28,28)$ images. These are the standard dimensions of the MNIST dataset, a generic dataset of handwritten digits that we will use later for our workflow.<jupyter_code># Interpolate spatial properties
ti = np.linspace(start=0, stop=1, num=28) #an array of 28 discrete points
XI, YI = np.meshgrid(ti,ti)               #a mesh of 28x28 discrete points
ZI = {}; ZI_s = {}                        

for i in features:
    # RBF interpolation
    ZI[i] = Rbf_interpolation(df_s['X'], df_s['Y'], df[i], function='thin_plate')(XI, YI)[::-1] 
    # Normalize our interpolated features 
    ZI_s[i]  = scaler.fit_transform(ZI[i])                                                           

for i in np.arange(len(features)):
    print(features[i]+': Shape={}, min={:.3f}, max={:.3f}'.format(ZI_s[features[i]].shape, 
                                                                  ZI_s[features[i]].min(), 
                                                                  ZI_s[features[i]].max()))
fig, axs = plt.subplots(2,4, figsize=(15,6))
for i in range(len(features)):
    axs[0,i].set_title(features[i]); axs[1,i].set_title('Interpolated '+features[i])
    # plot original data scatterplots
    im1 = axs[0,i].scatter(x=df_s['X'], y=df_s['Y'], s=8, c=df_s[features[i]], cmap=my_maps[i])
    fig.colorbar(im1, ax=axs[0,i])
    # plot interpolated images
    im2 = axs[1,i].imshow(ZI_s[features[i]], cmap=my_maps[i])
    fig.colorbar(im2, ax=axs[1,i])
    # remove ticks, set square box ratios
    for k in range(2):
        axs[k,i].set_xticks([]); axs[k,i].set_yticks([]); 
        axs[k,i].set_aspect('equal', adjustable='box')
plt.show();<jupyter_output><empty_output><jupyter_text>Select one of the subsurface features to be used in the remainder of the notebook for image reconstruction.<jupyter_code># Select one of the subsruface features to work with
feature_selected = 'AI'<jupyter_output><empty_output><jupyter_text>***  4. Advanced Dimensionality ReductionDimensionality Reduction is quite ubiqutuous in modern machine learning. PCA has been widely-studied and applied in theoretical and applied setting for data science, including reservoir characterization, modeling, and simulation. However, PCA assumes linearity and no outliers, while  SVD will simply decompose the matrix into left-and-right singular vectors and a diagonal singular value matrix. With this, we can project our 2D data onto the vectors and work in latent space.Another consideration is the idea of using the latent space for a generic dataset as the basis for reconstruction of a more complex dataset. For instance, using the SVD decomposition of the $60,000$ MNIST images, we can reconstruct our 2D subsurface maps from a latent represenation!![image-2.png](attachment:image-2.png)  4. Singular Value Decomposition (SVD)We start by implementing SVD on our MNIST dataset to obtain our singular values and right-and-left singular vectors. With this, we only select a subset such that we encode our matrices onto the latent space and work with a reduced dimensionality problem. Then we reconstruct our images from the inverse encoders using only $k$-dimensional latent vector, and analyze the results:Singular Value Decomposition is given by:        $$ X = U \Sigma V^* $$where for a given data matrix $X \in \mathbb{C}^{(n\times m)}$ we have:$U$ and $V$ are the left-singular $(n\times n)$ and right-singular $(m\times m)$ vectors, respectively, and $\Sigma$ is the non-negative, diagonal matrix $(n\times m)$ of ranked singular values, $\sigma_i = \Sigma_{ii}$. The amount of non-zero singular values equals the rank of $X$. Moreover, these singular values are sorted in descending order automatically. If $X$ is a real matrix, then $U$ and $V$ are guaranteed to be real orthogonal matrices; and SVD simply becomes: $X=U\Sigma V^T$. For a correlated dataset, we can compute an Economy, Truncated, or Compact-SVD such that we obtain:        $$ X \approx \hat{U} \hat{\Sigma} \hat{V}^* = \hat{X}$$where $\hat{\bullet}$ represents the first $k$ leading columns or block of the decomposition elements. Here, $\hat{U}$ is an $(m\times k)$ semi-unitary matrix and $V$ is an $(n\times k)$ semi-unitary matrix as well, such that: $U^*U = V^*V = I$, the identity matrix.Note that $k$ will fundamentally depend on the rank of the matrix $X$ such that $k\leq\min{(m,n)}$. Compact SVD performed by only removing the $0$ singular values will not results in lossy compression, but further dimensionality reduction by only keeping the leading $k$ singular values will result in some level of loss.For reconstruction using the latent space variables, it then follows that: $ \tilde{X}_{test} = \sum\limits_{i=1}^{k}\alpha_i\tilde{u}_i=\tilde{U}\alpha$, where $\alpha_i$ are the coefficients tht can be estimated to give the best MSE approximation. Then: $ \tilde{U}^*x_{test} = \tilde{U}^*\tilde{U}\alpha=\alpha \Longrightarrow \tilde{x}_{test}=\tilde{U}\tilde{U}^*x_{test}$.<jupyter_code># Perform Singular value decomsposition on our MNIST data (flat, centered)
U, S, VT = SVD(x_train_f_c, full_matrices=False)   #full_matrices=False -> economy SVD
S = np.diag(S)                                     #re-diagonalize S

print('Shapes: U={}, S={}, VT={}'.format(U.shape, S.shape, VT.shape))<jupyter_output>Shapes: U=(784, 784), S=(784, 784), VT=(784, 5000)<jupyter_text>With our transformation matrix, lets select and visualize the first few $k$ left-singular vectors (which will be used for our projection and prediction later).<jupyter_code># Define how many singular values/vectors we want to keep
k = 36<jupyter_output><empty_output><jupyter_text>We can see how the choice of retaining $k$ singular values value compares to the general shape of the singular values in our data matrix, in terms of the relative cumulative singular value sum.<jupyter_code>ks = [1, 10, 50, 100, 500]
S_ = np.cumsum(np.diag(S))/np.sum(np.diag(S))  # make it singular value "relative"

plt.figure()
plt.plot(S_, c='k', label='All singular values')
plt.scatter(ks, S_[ks], s=50, c='r', label='Sample singular values')
plt.hlines(S_[k], 0, 784, color='k', linestyle='--', linewidth=1)
plt.scatter(k, S_[k], c='b', marker='*', s=100, label='selected (k) singular value')
for j in ks:
    plt.text(j+25, S_[j], j, bbox=my_box)
    plt.text(300, S_[k], 'k='+str(k), bbox=my_box)
plt.title('Cumulative sum of singular values'); plt.grid('on')
plt.xlabel('Singular Value index'); plt.ylabel('Cumulative Singular Value Sum')
plt.legend(loc='lower right')
plt.show();
# Visualize the first "k" left-singular vectors
#reshape basis onto 28x28 images for visualization before plotting
U_ = np.reshape(U[:, :k].T, (-1, x_train.shape[1], x_train.shape[2], 1))
print('U_ shape:', U_.shape)
plot_sample_matrix(U_, my_cmap='cividis')
plt.suptitle('SVD (MNIST) Bases')
plt.show();<jupyter_output>U_ shape: (36, 28, 28, 1)<jupyter_text>We now project our $60,000$ MNIST digits onto the left-singular vector latent space $U$ and compute the transpose of the (pseudo) inverse of this matrix to obtain our latent representations.<jupyter_code># Project onto k basis
v = (np.linalg.pinv(U[:, :k]) @ x_train_f_c).T
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.scatter(v[:, 0], v[:, 1], s=30, c=cs, alpha=0.6)
plt.xlabel('$z_1$'); plt.ylabel('$z_2$'); 
plt.title('Latent Space Projections 1-2')
plt.subplot(1, 2, 2)
plt.scatter(v[:, 1], v[:, 2], s=30, c=cs, alpha=0.6)
plt.xlabel('$z_2$'); plt.ylabel('$z_3$'); 
plt.title('Latent Space Projections 2-3')
plt.show();<jupyter_output><empty_output><jupyter_text>With this latent representation, we can now reconstruct into our predicted images using the first $k$ basis and projected back with $V^T$ to form $\tilde{X}=\tilde{U}\tilde{\Sigma}\tilde{V}^T$.<jupyter_code># (TRAIN) Reconstruct using k basis
x_train_f_c_hat = U[:, :k] @ v.T                                                     #project
x_train_f_hat = x_train_f_c_hat + np.expand_dims(np.mean(x_train_f, axis=1), axis=1) #uncenter
x_train_hat = np.reshape(x_train_f_hat.T, [-1, 28, 28])                              #unflatten
num_rows, num_cols = 3, 3
plt.figure(figsize=(2*2*num_cols, 2*num_rows))
for i in range(num_rows*num_cols):
    plt.subplot(num_rows, 2*num_cols, 2*i+1)
    plt.grid(False); plt.xticks([]); plt.yticks([])
    plt.imshow(x_train[i], vmin=0, vmax=1, cmap='viridis')
    if i < 3: plt.title('Image')
    plt.subplot(num_rows, 2*num_cols, 2*i+2)
    plt.grid(False); plt.xticks([]); plt.yticks([])
    plt.imshow(x_train_hat[i], vmin=0, vmax=1, cmap='turbo')
    if i < 3: plt.title('Reconstruction')
plt.show();<jupyter_output><empty_output><jupyter_text>We now have predicted images $\hat{x}$ for our MNIST dataset. Essentially, the decoded images from a truncated latent represenation obtained from SVD.We will visualize these reconstructed images, and compare their quality by means of pixel-wise MSE and SSIM. Where:    $$ MSE = \frac{1}{n} \sum\limits_{i=1}^{n}(y_i-\hat{y}_i)^2$$and    $$ SSIM(x,y) = \frac{(2\mu_x\mu_y+c_1)(2\sigma_{xy}+c_2)}{(\mu_x^2+\mu_y^2+c_1)(\sigma^2_x+\sigma^2_y+c_2)}$$where $y_i$ are the true images and $\hat{y}_i$ are the reconstructed images for the MSE computation. On the other hand, we have that for SSIM, $x$ and $y$ are the two images to be compared, and $c_1 = (k_1L)^2$ and $c_2=(k_2L)^2$ are two variables to stabilize the division with weak denominators. Usually, $k_1=0.01$ and $k_2=0.03$, and $L=2^{(\bits/pixel)}-1$ typically.For MSE calculation, we will use the flattened, centered arrays $N \times M$ as opposed to the images $N \times (M,D)$ used in the SSIM calculation.<jupyter_code># Error Metrics for selected level of k
mse  = mean_squared_error(x_train_f_c, x_train_f_c_hat)
ssim = SSIM(x_train.squeeze(), x_train_hat)
print('MSE={:.3f} | SSIM={:.3f}'.format(mse,ssim))<jupyter_output>MSE=0.015 | SSIM=0.856<jupyter_text>A really low MSE and high SSIM! That is great considering we only kept $k$ singular values/vectors from our original data! *** **Subsurface SVD:**Next, we will use those $k$ singular values/vectors comptued from the truncated SVD decomposition of MNIST, and project our $28x28$ subsurface maps onto them. After projection, we will decode the latent variables using the same structure as before and obtain new reconstructions from the reduced-dimensionality maps.<jupyter_code>A_gsc_r_f   = np.reshape(ZI_s[feature_selected], (28*28, 1))                    #flatten
A_gsc_r_f_c = A_gsc_r_f - np.expand_dims(np.mean(x_train_f, axis=1), axis=1)    #center
print('Image shape:', A_gsc_r_f_c.shape)

# Plot the processed 2D map
plt.figure(figsize=(4, 4))
plt.imshow(np.reshape(A_gsc_r_f_c, (28, 28)), vmin=0, vmax=1,
           cmap=my_maps[features.index(feature_selected)])
plt.title('Original '+feature_selected+' Map'); plt.xticks([]); plt.yticks([])
plt.colorbar(shrink=0.85)
plt.show();<jupyter_output>Image shape: (784, 1)<jupyter_text>Try reconstructing the image from the latent variable of MNIST to a full 2D subsurface reservoir property map using a few different $k$'s, where $k$ is the size of the latent vector.Note that we project our image *A_gsc_r_f_c* onto the precomputed left-singular vectors of the MNIST matrix $\tilde{U}$. This is projecting our image onto the generic latent space $z_{MNIST}$.<jupyter_code>ks = [10, 150, 500, 784]

plt.figure(figsize=(len(ks)*5,4))

plt.subplot(1,len(ks)+1,1)
plt.imshow(ZI_s[feature_selected], cmap=my_maps[features.index(feature_selected)])
plt.title('Original '+feature_selected+' Map')
plt.colorbar(fraction=0.046, pad=0.04); plt.xticks([]); plt.yticks([])

for i in np.arange(len(ks)):
    v = (np.linalg.pinv(U[:, :ks[i]]) @ A_gsc_r_f_c).T  #project image onto latent space
    A_gsc_r_f_c_hat = U[:, :ks[i]] @ v.T
    A_gsc_r_f_hat   = A_gsc_r_f_c_hat + np.expand_dims(np.mean(x_train_f, axis=1), axis=1)
    A_gsc_r_hat     = np.reshape(A_gsc_r_f_hat.T, [-1, 28, 28])
    
    mse  = mean_squared_error(A_gsc_r_f_c, A_gsc_r_f_c_hat)
    ssim = SSIM(ZI_s[feature_selected], A_gsc_r_hat.squeeze())

    plt.subplot(1,len(ks)+1,i+2)
    img = plt.imshow(np.squeeze(A_gsc_r_hat), vmin=0, vmax=1,
                     cmap=my_maps[features.index(feature_selected)])
    plt.title(feature_selected+' Reconstruction: k='+ str(ks[i]))
    plt.xlabel('MSE={:.3f} | SSIM={:.3f}'.format(mse, ssim))
    plt.colorbar(fraction=0.046, pad=0.04); plt.xticks([]); plt.yticks([])
    
plt.show();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12354663680992163, 'fraction_numerical': 0.0189044871242787, 'mean_word_length': 4.8013989507869095, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 82, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '334307', 'license': 'MIT', 'n_tokens_mistral': 8090, 'n_tokens_neox': 7617, 'n_words': 2429, 'path': 'ADR_SVD.ipynb', 'repo_name': 'misaelmmorales/Dimensionality-Reduction', 'size': '23237'}