<jupyter_start><jupyter_text>Hey Fili!  So this should hopefully give you some ideas for things to play with in Jupyter, Python, Pandas, etc.  I know that all of this is super new, but that's okay!   I'm definitely not going to be able to give a huge in-depth thing around this.. but I can share links!  What you're currently looking at is a [Jupyter Notebook](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html). This is an interactive IDE (integrated development environment) where you can slap in Python code and run it in individual cells, while also being able to document your work using the Markdown language! (That's what this cell is, actually.)[Markdown](https://www.ultraedit.com/company/blog/community/what-is-markdown-why-use-it.html) is a language that helps with basic text formatting, similar to a word processor. It's very popular among developers because of its ease of use.  Anyway, I'm using this as a way to help guide some of your learning and give you some real-world examples of what you can do with Jupyter, Python.. that kind of stuff.<jupyter_code># This is an actual 'code' cell where you can run Python code. This is a comment.
# If you hold Left Ctrl and hit Enter, you'll run the actual code below.

print('Press Ctrl + Enter to make me print a line.')<jupyter_output><empty_output><jupyter_text>The great thing about Jupyter is that you can split your code into these cells and run them individually. Anything you run stays in memory until you restart the kernel. So this lets you develop code in steps!<jupyter_code># For example.. I'll create a variable that holds your name! If you run this cell (Ctrl + Enter) the variable will be created.

name = 'Fili'
# And then when you run this cell, it will say hello with your name!

print(f'Hey there {name}! Glad you could make it!')
# Since things are stored in memory, you can continue to develop new cells without having to run the previous cells.
# In normal Python scripts, you would have to run the entire script in one go. 
# This lets you develop things pretty quick, and is why I like Jupyter so much for my work.

print(f'I can reuse the name variable ({name}) as much as I want without having to run previous cells again. Whee.')<jupyter_output><empty_output><jupyter_text>Normally you would have to install these packages in a new environment, but I took care of that already using the `requirements.txt` file. All of the imports below should work just fine! I included some notes about each one for more research but.. it's probably a ton of reading, haha.The imports below are some of my most commonly used tools.[What is Pandas?](https://towardsdatascience.com/a-quick-introduction-to-the-pandas-python-library-f1b678f34673)<jupyter_code>import pandas as pd
import numpy as np
import os
# os is a nifty operating system manipulation package. You can do a lot with it.
# I normally use it to load files or iterate through file folders on my computer.

# Like this! It'll print out the contents of the root folder for this cloud environment.
os.listdir()
# Which is cool, but not super helpful for what I normally do. What I would normally do looks more like this -

for file in os.listdir():
    # Do some stuff to the files, like load them to pandas (if they are CSV or Excel files, or even PDF files)
    # Here I'm just going to print them.. again.
    print(file)<jupyter_output><empty_output><jupyter_text>Pandas is pretty much my de-facto tool for *everything*. Literally everything. It loads things to a familiar spreadsheet-style view and lets you manipulate it. There's so, so much I can share with this. But.. baby steps.Here's a quick example using some Gamestop stock market data I downloaded a while back.<jupyter_code># I load the file using pandas and save it to a variable named 'gamestop'
gamestop = pd.read_csv('gme-daily-ftd.csv')<jupyter_output><empty_output><jupyter_text>So.. uh.. nothing happened? Well, it did! We just didn't actually do anything with the variable after creating it. Here's a few things you could do, try copying them into the cell below and running them one at a time!gamestop.info()gamestop.head()gamestop.describe()gamestop['QUANTITY (FAILS)'].describe()These two below show one of the common things I actually do with a lot of data I work with, especially stuff I have to load from Excel or CSV. The 'DATETIME' column SHOULD be a date or a time, but it's an 'object' data type. Which you can see with gamestop.info() above. This means that normal date-type operations can't be easily done to it, and it behaves like a string instead. You can easily convert DATETIME to the right data type using pandas.gamestop['DATETIME']pd.to_datetime(gamestop['DATETIME']gamestop['DATETIME_FIXED'] = pd.to_datetime(gamestop['DATETIME']These three commands will (1) show a few rows of the DATETIME field (2) convert those rows to an actual datetime data type and (3) create a new column named DATETIME_FIXED that has the corrected data type fieldHopefully this is a nice primer for what you might be able to do with Pandas.I included a second CSV file that you can load and play around with. Can you find a way to answer these questions without actually opening the CSV file, but instead parsing it with pandas?1. How many rows of data are there in the CSV?2. What's the highest OPEN and CLOSE values in the file?3. What's the earliest date in the file?<jupyter_code># CSV filename is 'GME.csv'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06421014228383802, 'fraction_numerical': 0.0027362276541408243, 'mean_word_length': 4.538383838383838, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '51109', 'license': 'MIT', 'n_tokens_mistral': 1572, 'n_tokens_neox': 1495, 'n_words': 830, 'path': 'Fili_Initial.ipynb', 'repo_name': 'Evernite/Teaching_Environment', 'size': '5482'}