<jupyter_start><jupyter_text>dislib tutorialThis tutorial will show the basics of using [dislib](https://dislib.bsc.es). RequirementsApart from dislib, this notebook requires [PyCOMPSs 2.5](https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/). SetupFirst, we need to start an interactive PyCOMPSs session:<jupyter_code>import pycompss.interactive as ipycompss
ipycompss.start(graph=True, monitor=1000)<jupyter_output><empty_output><jupyter_text>Next, we import dislib and we are all set to start working!<jupyter_code>import dislib as ds<jupyter_output><empty_output><jupyter_text>Distributed arraysThe main data structure in dislib is the distributed array (or ds-array). These arrays are a distributed representation of a 2-dimensional array that can be operated as a regular Python object. Usually, rows in the array represent samples, while columns represent features.To create a random array we can run the following NumPy-like command:<jupyter_code>x = ds.random_array(shape=(500, 500), block_size=(100, 100))
print(x.shape)
x<jupyter_output><empty_output><jupyter_text>Now `x` is a 500x500 ds-array of random numbers stored in blocks of 100x100 elements. Note that `x` is not stored in memory. Instead, `random_array` generates the contents of the array in tasks that are usually executed remotely. This allows the creation of really big arrays.The content of `x` is a list of `Futures` that represent the actual data (wherever it is stored).To see this, we can access the `_blocks` field of `x`:<jupyter_code>x._blocks[0][0]<jupyter_output><empty_output><jupyter_text>`block_size` is useful to control the granularity of dislib algorithms.To retrieve the actual contents of `x`, we use `collect`, which synchronizes the data and returns the equivalent NumPy array:<jupyter_code>x.collect()<jupyter_output><empty_output><jupyter_text>Another way of creating ds-arrays is using array-like structures like NumPy arrays or lists:<jupyter_code>x1 = ds.array([[1, 2, 3], [4, 5, 6]], block_size=(1, 3))
x1<jupyter_output><empty_output><jupyter_text>Distributed arrays can also store sparse data in CSR format:<jupyter_code>from scipy.sparse import csr_matrix

sp = csr_matrix([[0, 0, 1], [1, 0, 1]])
x_sp = ds.array(sp, block_size=(1, 3))
x_sp<jupyter_output><empty_output><jupyter_text>In this case, `collect` returns a CSR matrix as well:<jupyter_code>x_sp.collect()<jupyter_output><empty_output><jupyter_text>Loading dataA typical way of creating ds-arrays is to load data from disk. Dislib currently supports reading data in CSV and SVMLight formats like this:<jupyter_code>x, y = ds.load_svmlight_file("../tests/files/libsvm/1", block_size=(20, 100), n_features=780, store_sparse=True)

print(x)

csv = ds.load_txt_file("../tests/files/csv/1", block_size=(500, 122))

print(csv)<jupyter_output><empty_output><jupyter_text>SlicingSimilar to NumPy, ds-arrays support the following types of slicing:(Note that slicing a ds-array creates a new ds-array)<jupyter_code>x = ds.random_array((50, 50), (10, 10))<jupyter_output><empty_output><jupyter_text>Get a single row:<jupyter_code>x[4]<jupyter_output><empty_output><jupyter_text>Get a single element:<jupyter_code>x[2, 3]<jupyter_output><empty_output><jupyter_text>Get a set of rows or a set of columns:<jupyter_code># Consecutive rows
print(x[10:20])

# Consecutive columns
print(x[:, 10:20])

# Non consecutive rows
print(x[[3, 7, 22]])

# Non consecutive columns
print(x[:, [5, 9, 48]])<jupyter_output><empty_output><jupyter_text>Get any set of elements:<jupyter_code>x[0:5, 40:45]<jupyter_output><empty_output><jupyter_text>Other functionsApart from this, ds-arrays also provide other useful operations like `transpose` and `mean`:<jupyter_code>x.mean(axis=0).collect()
x.transpose().collect()<jupyter_output><empty_output><jupyter_text>Machine learning with dislibDislib provides an estimator-based API very similar to [scikit-learn](https://scikit-learn.org/stable/). To run an algorithm, we first create an estimator. For example, a K-means estimator:<jupyter_code>from dislib.cluster import KMeans

km = KMeans(n_clusters=3)<jupyter_output><empty_output><jupyter_text>Now, we create a ds-array with some blob data, and fit the estimator:<jupyter_code>from sklearn.datasets import make_blobs

# create ds-array
x, y = make_blobs(n_samples=1500)
x_ds = ds.array(x, block_size=(500, 2))

km.fit(x_ds)<jupyter_output><empty_output><jupyter_text>Finally, we can make predictions on new (or the same) data:<jupyter_code>y_pred = km.predict(x_ds)
y_pred<jupyter_output><empty_output><jupyter_text>`y_pred` is a ds-array of predicted labels for `x_ds` Let's plot the results<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt


centers = km.centers

# set the color of each sample to the predicted label
plt.scatter(x[:, 0], x[:, 1], c=y_pred.collect())

# plot the computed centers in red
plt.scatter(centers[:, 0], centers[:, 1], c='red')<jupyter_output><empty_output><jupyter_text>Note that we need to call `y_pred.collect()` to retrieve the actual labels and plot them. The rest is the same as if we were using scikit-learn. Now let's try a more complex example that uses some preprocessing tools.First, we load a classification data set from scikit-learn into ds-arrays. Note that this step is only necessary for demonstration purposes. Ideally, your data should be already loaded in ds-arrays.<jupyter_code>from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

x, y = load_breast_cancer(return_X_y=True)

x_train, x_test, y_train, y_test = train_test_split(x, y)

x_train = ds.array(x_train, block_size=(100, 10))
y_train = ds.array(y_train.reshape(-1, 1), block_size=(100, 1))

x_test = ds.array(x_test, block_size=(100, 10))
y_test = ds.array(y_test.reshape(-1, 1), block_size=(100, 1))<jupyter_output><empty_output><jupyter_text>Next, we can see how support vector machines perform in classifying the data. We first fit the model (ignore any warnings in this step):<jupyter_code>from dislib.classification import CascadeSVM

csvm = CascadeSVM()

csvm.fit(x_train, y_train)<jupyter_output><empty_output><jupyter_text>and now we can make predictions on new data using `csvm.predict()`, or we can get the model accuracy on the test set with:<jupyter_code>score = csvm.score(x_test, y_test)<jupyter_output><empty_output><jupyter_text>`score` represents the classifier accuracy, however, it is returned as a `Future`. We need to synchronize to get the actual value:<jupyter_code>from pycompss.api.api import compss_wait_on

print(compss_wait_on(score))<jupyter_output><empty_output><jupyter_text>The accuracy should be around 0.6, which is not very good. We can scale the data before classification to improve accuracy. This can be achieved using dislib's `StandardScaler`.The `StandardScaler` provides the same API as other estimators. In this case, however, instead of making predictions on new data, we transform it:<jupyter_code>from dislib.preprocessing import StandardScaler

sc = StandardScaler()

# fit the scaler with train data and transform it
scaled_train = sc.fit_transform(x_train)

# transform test data
scaled_test = sc.transform(x_test)<jupyter_output><empty_output><jupyter_text>Now `scaled_train` and `scaled_test` are the scaled samples. Let's see how SVM perfroms now.<jupyter_code>csvm.fit(scaled_train, y_train)
score = csvm.score(scaled_test, y_test)
print(compss_wait_on(score))<jupyter_output><empty_output><jupyter_text>The new accuracy should be around 0.9, which is a great improvement!  Close the sessionTo finish the session, we need to stop PyCOMPSs:<jupyter_code>ipycompss.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1020646669263732, 'fraction_numerical': 0.018179457213348914, 'mean_word_length': 6.956611570247934, 'pattern_counts': {'":': 0, '<': 105, '<?xml version=': 0, '>': 105, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '143751', 'license': 'Apache-2.0', 'n_tokens_mistral': 2720, 'n_tokens_neox': 2577, 'n_words': 772, 'path': 'notebooks/tutorial.ipynb', 'repo_name': 'sravyasri/dislib', 'size': '7701'}