<jupyter_start><jupyter_text>It's worth remembering that the first PC is mostly driven by standard deviation while all variables contribute to the second.  Cluster stability - series temporal stabilityFor the purpose of this notebook, cluster stability is defined as the tendency of series to belong to the same cluster over time.  A baseline clustering algorithm and clusters are defined in this section and the series are classified at different times using the same model.  Two experiments are run: a cumulative time experiment (i.e. each period starts at day1), and a rolling experiment. Both experiments are run on timeseries and visualised on the principal components.**NOTE**: This is conducted well avare of [Keogh at al.](http://www.cs.ucr.edu/~eamonn/meaningless.pdf). Results will have to be examined with great care. Training the modelThe clustering model is a Timeseries KMeans using [soft DTW](https://arxiv.org/pdf/1703.01541.pdf) as distance metric.<jupyter_code>from tslearn import utils, clustering
%%time
# training can take up to 30 minutes, depending on hardware resources
seed(1)
series = utils.to_time_series_dataset(data.T.values)
km = clustering.TimeSeriesKMeans(n_clusters=15, metric="softdtw", max_iter=10)
km.fit(series)<jupyter_output>571448.112 --> 631432.750 --> 642338.522 --> 646458.893 --> 646657.061 --> 646932.626 --> 647203.437 --> 647448.775 --> 647462.378 --> 647647.453 --> 
CPU times: user 16min 39s, sys: 24.5 s, total: 17min 3s
Wall time: 17min 2s<jupyter_text>Cumulative time period<jupyter_code>from matplotlib import animation, rc
from IPython.display import HTML
sys.path.append("../../iz4vve/utils")
import tools
cluster_assignments = list()
cumulative = None

for chunk in tqdm.tqdm_notebook(tools.chunker(data, 90)):
    if cumulative is None:
        cumulative = chunk.values
    else:
        cumulative = np.concatenate([cumulative, chunk.values])
    clusters = km.predict(cumulative.T)
    cluster_assignments.append(clusters)
pd.DataFrame(cluster_assignments).plot(legend=False)
plt.title("Series cluster evolution")
plt.show()
changes = pd.DataFrame(cluster_assignments).diff().abs()
print("how many times series change cluster:")
for c in list(changes):
    print(f"{c}  -->  {np.count_nonzero(changes[c].values)}")
cluster_assign = pd.DataFrame(cluster_assignments)
cluster_assign.shape
import ipywidgets as widgets
@widgets.interact(x=range(1, cluster_assign.shape[0]))
def g(x):
    _df = cluster_assign.iloc[x]
    plt.scatter(*list(zip(*pcaed)), c=_df, cmap="rainbow")
    plt.title(f"Cluster at time {x}")
    plt.show()<jupyter_output><empty_output><jupyter_text>The graph above shows some slight changes for cluster assignments over time, but clusters appear overall stable.  Rolling time period<jupyter_code># %%time
cluster_assignments = list()

for chunk in tqdm.tqdm_notebook(tools.chunker(data, 90)):
    clusters = km.predict(chunk.T.values)
    cluster_assignments.append(clusters)
pd.DataFrame(cluster_assignments).plot(legend=False)
plt.title("Series cluster evolution")
plt.show()
changes = pd.DataFrame(cluster_assignments).diff().abs()
print("how many times series change cluster:")
for c in list(changes):
    print(f"{c}  -->  {np.count_nonzero(changes[c].values)}")
cluster_assign = pd.DataFrame(cluster_assignments)
cluster_assign.shape
import ipywidgets as widgets
@widgets.interact(x=range(1, cluster_assign.shape[0]))
def g(x):
    _df = cluster_assign.iloc[x]
    plt.scatter(*list(zip(*pcaed)), c=_df, cmap="rainbow")
    plt.title(f"Cluster at time {x}")
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11566332218506131, 'fraction_numerical': 0.03595317725752508, 'mean_word_length': 5.274475524475524, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '75168', 'license': 'MIT', 'n_tokens_mistral': 1232, 'n_tokens_neox': 1112, 'n_words': 326, 'path': 'notebooks/4. Cluster temporal stability.ipynb', 'repo_name': 'iz4vve/dseclub-clustering', 'size': '3588'}