<jupyter_start><jupyter_text>TodayIntroduction- Electric Tesla ferry - guest lecture, 5th November. <NAME>.- Tools    - Cloud: Google Colaboratory    - Local: Anaconda Python    - Jupyter Notebook IDE- Resources    - https://kaggle.com    - https://github.com/frahlg/UBC-NAME    - Getting started Assignment- Merge .xlsx/.csv files, time-series- Explanatory statistics    - You are free to explore.- Filter data    - Based on your previous expertise. Think of what you need to filter.    - What is 'bad' data?- Visualise the data.    - Think histograms, line-plots and bar-charts. Use ```matplotlib```- Export data to new format, preferably both .xlsx and .hd5....- If time permits: ML ...<jupyter_code># This is is a comment . NOT CODE
import pandas as pd

# Files

# Note the "?raw=true"
# https://github.com/frahlg/UBC-NAME/blob/master/data/dataset_ship1/out_1.xlsx?raw=true


df = pd.read_excel('https://github.com/frahlg/UBC-NAME/blob/master/data/dataset_ship1/out_0.xlsx?raw=true',index_col=0)



# If the file is uploaded in the same folder as the Notebook.
# That is locally or if you upload via the web interface

df = pd.read_excel('out_0.xlsx')
# You could write several lines of code ...

df0 = pd.read_excel(f'https://github.com/frahlg/UBC-NAME/blob/master/data/dataset_ship1/out_0.xlsx?raw=true',index_col=0)
df1 = pd.read_excel(f'https://github.com/frahlg/UBC-NAME/blob/master/data/dataset_ship1/out_1.xlsx?raw=true',index_col=0)

# and so on ...

# But what if you have hundreds of files?


# Behold - the FOR loop!
for i in range(7):
    print(i)
all_data = pd.DataFrame()

for n in range(7):
    df = pd.read_excel(f'https://github.com/frahlg/UBC-NAME/blob/master/data/dataset_ship1/out_{n}.xlsx?raw=true',index_col=0)
    all_data = all_data.append(df)
all_data
all_data.to_excel('my_out_data.xlsx')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13203463203463203, 'fraction_numerical': 0.010822510822510822, 'mean_word_length': 3.8403141361256545, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '594487', 'license': 'MIT', 'n_tokens_mistral': 697, 'n_tokens_neox': 671, 'n_words': 165, 'path': 'notebooks/oct-22.ipynb', 'repo_name': 'hasnat2379/578-Data', 'size': '1859'}