<jupyter_start><jupyter_text>To summarize the contents of this notebook:1. I have read in the Train and Test datasets and reduced their sizes dramatically.2. I have also written them as pickle files into the working directory, which means loading them later takes a very short amount of time.3. I confirmed that there are 600 satellites in Train and their true and simulated kinematics were all recorded in January 2014. Likewise, there are 300 satellites in Test and their simulated kinematics were all recorded in February 2014.4. I inserted a new column called absolute time (abs_time) as a record of time that has passed since 00:00:00 1-Jan-2014, for both Train and Test.5. I extracted Satellite-1 data (the second satellite out of 600) and wrote them into pickle files. Not entirely sure what to do next, but this is my overview of the next few objectives:- Study time series analysis, forecasting and support vector machines.- Read more papers about the subject of predicting RSO (Resident Space Objects) trajectories using a ML approach.- Carry out a literature review (2000 words should be fine).- Use Sat_1 data as a small sample to learn to apply forecasting techniques before moving on to the remaining 599.- Have to submit predictions to the IDAO portal before the deadline on 12 Feb 2020**<NAME> 9:08am 22-Jan-2020**  20-Jan-2020  File Size Reduction<jupyter_code>import os
import numpy as np
import pandas as pd
import datetime as dt
%%time
train = pd.read_csv('train.csv')
# refer to https://www.kaggle.com/frankherfert/tips-tricks-for-working-with-large-datasets
train.info(memory_usage='deep')
# epoch column is an object that uses up the most memory
train.memory_usage(deep=True) * 1e-6
# convert epoch to datetime
print("size before:", train["epoch"].memory_usage(deep=True) * 1e-6)
train["epoch"] = pd.to_datetime(train["epoch"], infer_datetime_format=True)
print("size after: ", train["epoch"].memory_usage(deep=True) * 1e-6)
train.memory_usage(deep=True).sum() * 1e-6
train.head()
# save to a pickle file 
train.to_pickle("train.pkl")
# compare sizes
print("train.csv:", os.stat('train.csv').st_size * 1e-6)
print("train.pkl:", os.stat('train.pkl').st_size * 1e-6)
# delete train dataset
del train

train = pd.read_pickle('train.pkl')
# get min and max values for integer columns; ljust is left justify
for col in train[['id','sat_id']]:
    print(col.ljust(10), f"min:{train[col].min()}".ljust(10), f"max:{train[col].max()}")
# python by default uses 64 bit integers, so convert them to unsigned 32bit or 16bit integers
# DO NOT convert float64 to lower types because less decimals mean less accuracy
train["id"] = pd.to_numeric(train["id"], downcast="unsigned")
train["sat_id"] = pd.to_numeric(train["sat_id"], downcast="unsigned")
train[['id','sat_id']].memory_usage(deep=True) * 1e-6
train.info()
train.memory_usage(deep=True) * 1e-6
# smallest possible size made into a pickle file
train.to_pickle('train_initial.pkl')<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code>%%time
train = pd.read_pickle('train_initial.pkl')
train.head()
# explore epoch column; extract years, months, days
epoch_year = train['epoch'].dt.year
epoch_month = train['epoch'].dt.month
epoch_day = train['epoch'].dt.day
# only one unique year 2014
epoch_year.unique()
# only one unique month January
epoch_month.unique()
# all 31 days of January
epoch_day.unique()<jupyter_output><empty_output><jupyter_text>Based on this information, all 600 satellites' kinematics were recorded only in January 2014<jupyter_code># do all satellites start being recorded at 00:00:00 1 Jan 2014?
first_indices = []   # id of first observation of satellite e.g. Sat_0 is 0, Sat_1 is 958
initial_records = [] #
for sat_no in train['sat_id'].unique():
    first_id = train['sat_id'].searchsorted(sat_no)
    first_indices.append(first_id[0])
    initial_records.append(train.iloc[first_id[0],1])
# yes
pd.Series(initial_records).unique()
first_indices[1]
# calculate absolute difference in time since the initial record 
# insert that next to the epoch column
train.insert(2, 'abs_time', train['epoch']-train['epoch'][0])
train.head()
# read the papers linked in the contest page; SOTA = State of the Affairs
train.to_pickle('train.pkl')<jupyter_output><empty_output><jupyter_text>21-Jan-2020<jupyter_code># make a separate dataset to only look at satellite 1 (second one in the dataset)
train1 = train.iloc[first_indices[1]:first_indices[2],:]
train1.head()
train1.tail(10)
# save that to a pickle file
train1.to_pickle('train1.pkl')
train1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2108 entries, 958 to 3065
Data columns (total 15 columns):
id        2108 non-null uint32
epoch     2108 non-null datetime64[ns]
sat_id    2108 non-null uint16
x         2108 non-null float64
y         2108 non-null float64
z         2108 non-null float64
Vx        2108 non-null float64
Vy        2108 non-null float64
Vz        2108 non-null float64
x_sim     2108 non-null float64
y_sim     2108 non-null float64
z_sim     2108 non-null float64
Vx_sim    2108 non-null float64
Vy_sim    2108 non-null float64
Vz_sim    2108 non-null float64
dtypes: datetime64[ns](1), float64(12), uint16(1), uint32(1)
memory usage: 226.5 KB<jupyter_text>Doing the same for Test<jupyter_code>%%time
test = pd.read_csv('test.csv')
test.info(memory_usage='deep')
test.memory_usage(deep=True) * 1e-6
test['epoch'] = pd.to_datetime(test['epoch'], infer_datetime_format=True)
test['id'] = pd.to_numeric(test['id'], downcast='unsigned')
test['sat_id'] = pd.to_numeric(test['sat_id'], downcast='unsigned')
test.info(memory_usage='deep')
test.memory_usage(deep=True) * 1e-6
# can I use first_indices also for test data to extract Sat_1 data? 
# no i cant because test data is data in february, which is a continuation from train data in januray
# first observation ID of Test_Sat_1 is 3927
test.head()
# last observation ID of Train_Sat_1 is  3926
train.iloc[first_indices[1]:first_indices[2],:].tail()<jupyter_output><empty_output><jupyter_text>22-Jan-2020<jupyter_code>first_indices_test = []
for sat_no in test['sat_id'].unique():
    first_id = test['sat_id'].searchsorted(sat_no)
    first_indices_test.append(first_id[0])
first_indices_test[1]
test.iloc[1900]
test.iloc[1901]
test.head()
len(test['sat_id'].unique())
test.iloc[:1901,:].tail()
train['epoch'][0]
# insert abs_time for test data, calculated since 1 Jan 2014
test.insert(3, 'abs_time', test['epoch']-train['epoch'][0])
test.head()
# test is finally ready to be written as pickle file
test.to_pickle('test_initial.pkl')
del test
%%time
test = pd.read_pickle('test_initial.pkl')
# extract Sat_1 test data and write to pickle file
test1 = test.iloc[:first_indices_test[1],:]
test1.to_pickle('test1.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1149374540103017, 'fraction_numerical': 0.04621044885945548, 'mean_word_length': 5.447817836812145, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '86088', 'license': 'MIT', 'n_tokens_mistral': 2407, 'n_tokens_neox': 2220, 'n_words': 709, 'path': '1. IDAO 2020 .ipynb', 'repo_name': 'ismaildawoodjee/Forecasting-Satellite-Trajectories-IDAO-2020', 'size': '6805'}