<jupyter_start><jupyter_text>Exploratory Data Analysis Before we start training models and running inference, let's take a brief look in the contents of the dataset, containing telemetry from moontracers deployed all around the world.Data files have been aggregated and anonymized for your security and convenience.  Loading and exploring data with Pandas The pandas library offers tools to effectively load, manipulate and analyse data.<jupyter_code>!pip install --upgrade pip
!pip install --upgrade pandas
import pandas as pd<jupyter_output><empty_output><jupyter_text>A pandas DataFrame is a table-like data structure used to store and manipulate data in memory.Let's create one from a CSV file on disk.<jupyter_code>filename = "moontracer-dataset.csv.gz"
data = pd.read_csv(filename, 
                   parse_dates = ["timestamp"], 
                   dtype = {"device_id":"string"}, 
                   index_col = "timestamp")
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 4496828 entries, 2020-02-22 23:59:59 to 2020-02-25 22:03:11
Data columns (total 3 columns):
 #   Column         Dtype 
---  ------         ----- 
 0   device_id      string
 1   motor_peak_mA  int64 
 2   battery        int64 
dtypes: int64(2), string(1)
memory usage: 137.2 MB<jupyter_text>Let's look at the first records to see if data as lodad correctly:<jupyter_code>data.head()<jupyter_output><empty_output><jupyter_text>And basic statistics for each column:<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Let's take a loook at the data count per device:<jupyter_code>data.groupby("device_id").count()
%store data<jupyter_output>Stored 'data' (DataFrame)<jupyter_text>Amazon Web Services In this workshop we'll use AWS services to store, process and visualize data. Let's ensure we can use the AWS command line interface by creating a S3 bucket and uploading our dataset to it.<jupyter_code>import random
import string 

rand_id = ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))
%store rand_id
bucket = "mt-ml-workshop-{}".format(rand_id)
%store bucket
bucket
!aws s3 mb "s3://{bucket}"
!aws s3 cp "moontracer-dataset.csv.gz" "s3://{bucket}/"
!aws s3 ls "s3://{bucket}"<jupyter_output>2021-05-10 17:02:53   58662143 moontracer-dataset.csv.gz
{'fraction_non_alphanumeric': 0.09930615784908933, 'fraction_numerical': 0.03512575888985256, 'mean_word_length': 4.479809976247031, 'pattern_counts': {'":': 1, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '39111', 'license': 'MIT', 'n_tokens_mistral': 799, 'n_tokens_neox': 713, 'n_words': 232, 'path': 'mt-eda.ipynb', 'repo_name': 'CaravanaCloud/moontrace-workshop', 'size': '2306'}