<jupyter_start><jupyter_text>this data has some null values because the desired number of rows must equal = `144(samples perday)*6259(grids)*62(days) = 55 880 652`  Resampling Data every 10 mins:<jupyter_code>df = pd.read_csv("full_grid.csv")
df = df.reset_index()
df["time"] = pd.to_datetime(df["time"])
df = df.set_index("time")
df = df.groupby("grid_square")
df_groups = list(df.groups.keys())
final_df = pd.DataFrame()
for grid in df_groups:
    df_g = df.get_group(grid)
    df_g = df_g.resample("10T", origin="2013-11-1 00:00:00").sum()
    final_df = pd.concat([final_df,df_g])
    print("============= grid {} resampled =====".format(grid))
final_df.drop(columns = ["index"], inplace = True)
final_df.to_csv("resampled_full_grid.csv", index = True)<jupyter_output>============= grid 38 resampled =====
============= grid 39 resampled =====
============= grid 40 resampled =====
============= grid 154 resampled =====
============= grid 155 resampled =====
============= grid 156 resampled =====
============= grid 157 resampled =====
============= grid 158 resampled =====
============= grid 159 resampled =====
============= grid 160 resampled =====
============= grid 271 resampled =====
============= grid 272 resampled =====
============= grid 273 resampled =====
============= grid 274 resampled =====
============= grid 275 resampled =====
============= grid 276 resampled =====
============= grid 277 resampled =====
============= grid 278 resampled =====
============= grid 285 resampled =====
============= grid 286 resampled =====
============= grid 288 resampled =====
============= grid 289 resampled =====
============= grid 388 resampled =====
============= grid 389 resampled =====
============= grid 390 resampled =====
============= grid 391 resam[...]<jupyter_text>Validating the resampling :<jupyter_code>final_df.shape
len(df_groups)
final_df.index[-1]
unique_times = final_df.index.value_counts()
unique_times[unique_times != len(df_groups)]
len(df.index)
final_df.head()
grids_list_groups
grids_list_groups = list(df.groups.keys())<jupyter_output><empty_output><jupyter_text>Converting the data to 2d snapshot for every time interval.<jupyter_code>time_steps = final_df.reset_index()['time'].unique()
grids = final_df.reset_index()['grid_square'].unique()

print("we have {} time steps, and {} grid sqaures".format(len(time_steps), len(grids)))
# # =========== 2D Array ====================================
# # Define the new data shape
# # time_steps = full_data.reset_index()['time'].unique()
# # grids = full_data.reset_index()['grid_square'].unique()

# ## Convert the pandas dataframe to a numpy array with shape()
# # Each sample in the 4-D array is an image represent one snapshot taken at time-step
# # to the city, each value represent the value of one grid at this time-step.
# data = np.zeros(dtype=np.float64, shape=(len(time_steps), len(grids)//98, len(grids)//117, 1))

# for i, j in enumerate(time_steps):
#     data[i, :, :, :] = final_df.loc[j].to_numpy().reshape(98, 117, 1)

# # save to npy file
# np.save('full_grid_resampled.npy', data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2720564645492461, 'fraction_numerical': 0.04074430542188001, 'mean_word_length': 6.31924882629108, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '182999', 'license': 'MIT', 'n_tokens_mistral': 1136, 'n_tokens_neox': 990, 'n_words': 243, 'path': 'new-final_data_preperation.ipynb', 'repo_name': 'tarekmoha/Telecom_traffic_forcasting', 'size': '3117'}