<jupyter_start><jupyter_text>something has gone wrong.  There should be 2 peaks just as before.<jupyter_code>pivoted.index<jupyter_output><empty_output><jupyter_text>hey, wait, there are onlyu 12 hours.  Not 24.<jupyter_code>np.unique(df.index.time)
!head -24 fremont.csv
pivoted.shape
X = pivoted.fillna(0).T.values #this converts it to an array
X.shape
pivoted.info()
np.info(X)<jupyter_output>class:  ndarray
shape:  (2250, 24)
strides:  (8, 18000)
itemsize:  8
aligned:  True
contiguous:  False
fortran:  True
data pointer: 0x21e42c99f10
byteorder:  little
byteswap:  False
type: float64<jupyter_text>Transposing the dataframe... each day is an observation which consists of 24 hours. What are the days in relationship to each other.Do PCA on it. To reduce the dimensionality.<jupyter_code>from sklearn.decomposition import PCA
X2 = PCA(2, svd_solver='full').fit_transform(X)

X2.shape
import matplotlib.pyplot as plt
plt.scatter(X2[:, 0], X2[:, 1])<jupyter_output><empty_output><jupyter_text>from the dimensionality reduction, it looks like we have two different days.<jupyter_code>from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(2)
gmm.fit(X)
labels = gmm.predict(X)
labels
plt.scatter(X2[:, 0], X2[:, 1], c=labels, cmap='rainbow')
plt.colorbar()<jupyter_output><empty_output><jupyter_text>what is going on within each of these clusters?Where labels is the red cluster, 0<jupyter_code>pivoted.T[labels == 0].T.plot(legend=False, alpha=0.1)
#shows non-commute rides
pivoted.T[labels == 1].T.plot(legend=False, alpha=0.1)<jupyter_output><empty_output><jupyter_text>Are they really week days and week ends?<jupyter_code>pivoted.columns<jupyter_output><empty_output><jupyter_text>we want to convert the days to days of the week.<jupyter_code>pd.DatetimeIndex(pivoted.columns)
pd.DatetimeIndex(pivoted.columns).dayofweek
dow = pd.DatetimeIndex(pivoted.columns).dayofweek
plt.scatter(X2[:, 0], X2[:, 1], c=dow, cmap='rainbow')
plt.colorbar()
dates = pd.DatetimeIndex(pivoted.columns)
dates[(labels == 0) & (dow < 5)]
#this ends up showing holidays, xmas, days when people don't work<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12065727699530517, 'fraction_numerical': 0.027699530516431925, 'mean_word_length': 6.721014492753623, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '5932', 'license': 'MIT', 'n_tokens_mistral': 839, 'n_tokens_neox': 790, 'n_words': 195, 'path': 'unsupervised_analysis.ipynb', 'repo_name': 'toddherman/jupyter_workflow', 'size': '2130'}