<jupyter_start><jupyter_text>Beware of Python 2.x vs 3.x - This entire tutorial uses 3.x   Jupyter notebooks<jupyter_code>print("test") # press shift + enter to execute this statement
# esc + a to add a new cell above, esc + b to add a new cell below
# esc + dd to delete a cell
2+2
x = 2+2
print(x)<jupyter_output><empty_output><jupyter_text>General python/Jupyter notebook tricks.<jupyter_code>#import sys
#sys.version

#import matplotlib
#matplotlib.__version__
#dir(sys)
#help(sys.stdout)
#sys.stdout?
#sys.exit?
#Show contextual help<jupyter_output><empty_output><jupyter_text>Numpy arraysA multi-dimensional array manipulation library in PythonContains fast implementations in C for most  operations.<jupyter_code>import numpy as np
native_array = [1, 2, 3, 10]
numpy_array = np.array(native_array)
for i in range(len(native_array)):
    print("native_array[i]:",native_array[i]," numpy_array[i]:",numpy_array[i])<jupyter_output><empty_output><jupyter_text>Those look the same, but are they really the same?<jupyter_code>print(type(arr), type(a))
b = np.array([[1, 2, 3, 4], [5, 6, 7, 10.0]], dtype=np.float32)
print(b)
print(b.shape, b.dtype)<jupyter_output><empty_output><jupyter_text>Numpy generation functions<jupyter_code>print(np.zeros(3))
print(np.ones((3,3),dtype=np.float64))
print(np.linspace(1,10,10)) # start, stop, num
print(np.logspace(1,3,3)) # start, stop, num<jupyter_output><empty_output><jupyter_text>Numpy index manipulation<jupyter_code>x = np.linspace(1,10,10)
x, x[:2], x[2:], x[:-2], x[::2], x[::-2], x[::-1]<jupyter_output><empty_output><jupyter_text>Built in math functions<jupyter_code>np.mean(x), np.sin(x), x*2<jupyter_output><empty_output><jupyter_text>MatplotlibA matlab-like plotting library for static, animated and interactive plotting with python.<jupyter_code>import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
# generate data
x = np.linspace(-np.pi, np.pi, 1024)
cos, sin = np.cos(x), np.sin(x)

# plot the data 
plt.plot(x, cos)
plt.plot(x, sin)

# Set x limits
plt.xlim(-4.0, 4.0)

# Set y limits
plt.ylim(-1.0, 1.0)

fig, (ax1,ax2) = plt.subplots(2,1)
ax1.plot(x, cos)
ax2.plot(x, sin)<jupyter_output><empty_output><jupyter_text>Time/Life savers: os, pathlib, shutil, subprocess, sys and argparseos, pathlib - for performing system related tasks such as directory navigation, running a system command, or manipulating paths: switching directories, building file/pathnamesshutil - high level file operators - copy, move, own, delete etc.. sys - System specific parameters and functions specifically getting command line arguments, std I/Oargparse - parse commandline arguments easily<jupyter_code>import os
os.getcwd()
#os.chdir("FundiPythonTutorial")
#os.chdir("..")
os.getcwd()
os.listdir()
os.system("touch test.txt")
from pathlib import Path
p = Path('.')
print(list(p.glob('**/*.py')))
os.system("echo 'Hello'")<jupyter_output><empty_output><jupyter_text>Okay, that returned success, but did it really do anything? We don't have any way of accessing the output from the called processes. This is where the `subprocess` packages comes handy<jupyter_code>import subprocess
import sys
import shlex
# Start a subprocess which is a bash shell, capture its output and error, and print
result = subprocess.run(
    ["/bin/bash", "-c", "echo 'Helo Output'"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
print("stdout:", result.stdout)
print("stderr:", result.stderr)
print("return code", result.returncode) # 0 means the process was executed successfully, non zero values mean a failure, with the number representing an error code. 
print("***********")

# same as above, but the echo statement modified to print to stderr instead
result = subprocess.run(
    ["/bin/bash", "-c", "echo 'Helo Error' 1>&2"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
print("stdout:", result.stdout, "stderr:", result.stderr, "return code: ", result.returncode)


# a simpler way to run the same command, and this time, the return code is explicity changed to 1, which denotes a failure of the command

result = subprocess.run(
    ["echo 'Simpler Hello'; exit 1;"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True
)
print("stdout:", result.stdout, "stderr:", result.stderr, "return code: ", result.returncode)<jupyter_output><empty_output><jupyter_text>Now, what happens if your subprocess runs longer?<jupyter_code>print("Starting the sub process")
result = subprocess.run(
    ["echo 'Hello, again!'; sleep 15"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True
)
print("stdout:", result.stdout, "stderr:", result.stderr, "return code: ", result.returncode)<jupyter_output><empty_output><jupyter_text>You see that while you run `subprocess.run` you are blocked and need to wait until the subprocess finishes execution before you can do anything else. This may not be ideal for most cases. Hence, there is a better way to run subprocess using `Popen` that opens a process in a separate thread, and allows you to communicate to it periodically using `poll()`<jupyter_code>cmd = shlex.split("/bin/bash loop.bash")
result = subprocess.run(cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("stdout:", result.stdout.decode(), "stderr:", result.stderr.decode(), "return code: ", result.returncode)
cmd = shlex.split("/bin/bash loop.bash")
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
# for c in iter(lambda: process.stdout.read(1), b''): 
#     sys.stdout.write(c)
    
while True:
    c = process.stdout.read(1)
    if c == b'':
        break;
    sys.stdout.write(c)
    
sys.stdout.write("Done.")<jupyter_output><empty_output><jupyter_text>What is more versatile is that subprocess allows to communicate to a process multiple times through its std I/O pipes. Let us create a simple Jupyter notebook ourselves, shall we?<jupyter_code>proc = subprocess.Popen(['python3', '-i'],
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)

# To avoid deadlocks: careful to: add \n to output, flush output, use
# readline() rather than read()
while True:
    inp = input()
    if inp == "exit":
        proc.stdin.close()
        proc.terminate()
        proc.wait(timeout=0.2)
        break
    inp = inp + "\n"
    proc.stdin.write(inp.encode('utf-8'))
    proc.stdin.flush()
    print(proc.stdout.readline())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13334373050530257, 'fraction_numerical': 0.012476606363069246, 'mean_word_length': 5.208131655372701, 'pattern_counts': {'":': 1, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '478420', 'license': 'MIT', 'n_tokens_mistral': 2202, 'n_tokens_neox': 2130, 'n_words': 614, 'path': 'Tutorial.ipynb', 'repo_name': 'vivekvenkris/FundiPythonTutorial', 'size': '6412'}