<jupyter_start><jupyter_text>Observe Signal ClassesIn this notebook, we'll do a very simple thing: Download the various file lists and use Pandas to see how many files we have for each signal classification. This is just a simple exercise to show you what are in the data  File Lists<jupyter_code>#If you are running this in IBM Apache Spark (via Data Science Experience)
base_url = 'https://dal05.objectstorage.service.networklayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b'

#ELSE, if you are outside of IBM:
base_url = 'https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b'

basic = '{}/simsignals_files/public_list_basic_v2_26may_2017.csv'.format(base_url)
primary_full = '{}/simsignals_files/public_list_primary_v2_full_1june_2017.csv'.format(base_url)
primary_medium = '{}/simsignals_files/public_list_primary_v2_medium_1june_2017.csv'.format(base_url)
primary_small = '{}/simsignals_files/public_list_primary_v2_small_1june_2017.csv'.format(base_url)

import pandas as pd
import io
import requests

s=requests.get(basic).content
basic_df=pd.read_csv(io.StringIO(s.decode('utf-8')))
basic_df.head()
basic_df.groupby(['SIGNAL_CLASSIFICATION']).count()
s=requests.get(primary_full).content
basic_df=pd.read_csv(io.StringIO(s.decode('utf-8')))
basic_df.groupby(['SIGNAL_CLASSIFICATION']).count()
s=requests.get(primary_medium).content
basic_df=pd.read_csv(io.StringIO(s.decode('utf-8')))
basic_df.groupby(['SIGNAL_CLASSIFICATION']).count()
s=requests.get(primary_small).content
basic_df=pd.read_csv(io.StringIO(s.decode('utf-8')))
basic_df.groupby(['SIGNAL_CLASSIFICATION']).count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12295578437310721, 'fraction_numerical': 0.041792852816474865, 'mean_word_length': 8.44, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '557626', 'license': 'Apache-2.0', 'n_tokens_mistral': 636, 'n_tokens_neox': 598, 'n_words': 98, 'path': 'tutorials/Step_2b_List_Of_Signal_Classes.ipynb', 'repo_name': 'lucask42/SETI-machine-learning', 'size': '1651'}