<jupyter_start><jupyter_text>What all things we require for machine learning with numpy?* indexing    * simple    * fancy* reshaping* visualizing dimensions* swapaxes* transpose* generating ones and zeros* generating random    * integers    * uniform    * normal* reversing - confusion* example to batch for prediction    ```python    b = np.expand_dims(a,axis=0)    c = np.expand_dims(a,axis=1)    ```* concatenation* np.where* shuffling  Creating arrays with numpy<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
a = np.ones((3, 3))
b = np.zeros((3, 3))
print(a)
print('\n')
print(b)
np.arange(1, 10).reshape((3, 3))
np.linspace(1, 100, 100)
a = np.ones((3, 3))
print(np.triu(a))
print('\n')
print(np.tril(a))
min = 1
max = 10
shape = (4, 5)
x = np.random.randint(min, max, shape)
x, x.shape
plt.hist(np.random.randint(1, 100, (20000, )))
plt.show()
plt.hist(np.random.rand(10000))
plt.show()
plt.hist(np.random.randn(10000))
plt.show()<jupyter_output><empty_output><jupyter_text>indexing & reversing<jupyter_code>x, x.shape
x[1, 2]
x[1:, ::2] # second row - end, 
        # column - all but in step 2
x1d = np.random.randint(1, 10, 10)
x1d
x1d[::-1]
x1d[-2::-1]<jupyter_output><empty_output><jupyter_text>reshaping<jupyter_code>x
x.shape
x.reshape(2, 10)
x.ravel()<jupyter_output><empty_output><jupyter_text>expanding dimensions for batch<jupyter_code>x
np.expand_dims(x[0], axis=0)<jupyter_output><empty_output><jupyter_text>Grid plot<jupyter_code>import matplotlib.image as mpimg
import matplotlib.pyplot as plt
def grid_plot(from_paths, size=(16,8), bs=16, ixs=[]):
    if not ixs: ixs = list(np.random.randint(0, len(from_paths) - 1, bs))
    batch_paths = [from_paths[i] for i in ixs]
    input = np.array([mpimg.imread(p) for p in batch_paths])
    no_images, height, width, channels = input.shape # this is nrows*ncols, height, width, channels

    ncols = int(np.sqrt(bs))
    nrows = no_images//ncols
    assert ncols*nrows == bs, "Batch size must be a perfect square"

    # we want nrows*height, ncols*width, channels
    # so,
    result = (input.reshape(nrows, ncols, height, width, channels) # changes to nrows, ncols, height, width, channels
            .swapaxes(1, 2) # (nrows(axis=0), ncols(axis=1), height(axis=2), width(axis=3), channels) becomes (nrows, height, ncols, width, channels)
            .reshape(height*nrows, width*ncols, channels)) # changes to the desired output
    plt.rcParams["figure.figsize"] = size
    plt.imshow(result)
from easy_vqa import get_test_image_paths
train_image_paths = get_test_image_paths()
grid_plot(train_image_paths)<jupyter_output><empty_output><jupyter_text>What all things we require for matplotlib ?* definitely subplots* figure and axis* rcParams* spaces between images* bar, line, scatter* scale* mpimg, imshow  subplots<jupyter_code>import matplotlib.pyplot as plt

fig = plt.figure(figsize=(4,4))
plt.subplots_adjust(top=2, right=2)

for i in range(4):
    ax = fig.add_subplot(2,2,i+1)
    ax.set_title(f'{i+1} Plot')

rows = 2
cols = 2
fig, axes = plt.subplots(rows, cols, figsize=(4,4))
plt.subplots_adjust(top=2, right=2)
for i in range(rows):
    for j in range(cols):
        axes[i][j].set_title(f"{i+j} location")<jupyter_output><empty_output><jupyter_text>adjust spacing between plots<jupyter_code>#       left  = 0.125  # the left side of the subplots of the figure
#       right = 0.9    # the right side of the subplots of the figure
#       bottom = 0.1   # the bottom of the subplots of the figure
#       top = 0.9      # the top of the subplots of the figure
#       wspace = 0.2   # the amount of width reserved for blank space between subplots
#       hspace = 0.2   # the amount of height reserved for white space between subplots<jupyter_output><empty_output><jupyter_text>bar, line, hist, scatter<jupyter_code>x = np.random.randn(200)
plt.hist(x)
plt.show()
departments = ["Science", "Math", "Physics"]
faculties_per_department = [100, 56, 50]

plt.bar(departments, faculties_per_department)
plt.show()
y = 10*x + 50
plt.plot(x, y)
plt.show()
plt.scatter(x, y+3*np.random.randn(len(x)))
plt.show()<jupyter_output><empty_output><jupyter_text>scaling plots<jupyter_code>plt.figure(figsize=(4, 4))
plt.scatter(x, y+3*np.random.randn(len(x)))
plt.show()<jupyter_output><empty_output><jupyter_text>images<jupyter_code>import matplotlib.image as mpimg
img = mpimg.imread("cat1.jpg")
plt.imshow(img)
plt.show()<jupyter_output><empty_output><jupyter_text>What all things we require for Pandas?|* reading data* selecting columns* understanding data* feather format* grouping* apply  reading data<jupyter_code>import pandas as pd
pd.read_csv
pd.read_json
pd.read_excel
pd.read_feather
! wget --header="Host: people.sc.fsu.edu" --header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36" --header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" --header="Accept-Language: en-US,en;q=0.9" --header="Referer: https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html" "https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv" -c -o 'cities.csv'
! dir
df = pd.read_csv('cities.csv')
df.head()
df.columns
people_dict = {
    "weight": pd.Series([68, 83, 112]),
    "birthyear": pd.Series([1984, 1985, 1992]),
    "children": pd.Series([0, 3]),
    "hobby": pd.Series(["Biking", "Dancing"]),
}
people = pd.DataFrame(people_dict)
people
df[['LatD', ' "LatM"', ' "LatS"']]
all_cities = pd.merge(left=city_loc, right=city_pop, on="city", how="outer")
df.columns
df[df.columns[df.columns.str.endswith('S"')]]
df[df['LatD'] > 30]<jupyter_output><empty_output><jupyter_text>grouping<jupyter_code>df = pd.DataFrame(
{
"A": ["foo", "bar", "foo"] * 3,
"B": ["one", "two", "three"] * 3,
"C": np.random.randint(1, 10, 9),
"D": np.random.randn(9),
}
)
df
df.groupby('A').sum()<jupyter_output><empty_output><jupyter_text>apply<jupyter_code>def get_even(val): 
    if val%2==0: return val
    else: return -1
df['C'].apply(get_even)
import numpy as np
np.where(df['C']%2 ==0, df['C'], -1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16107600064819316, 'fraction_numerical': 0.032571706368497814, 'mean_word_length': 5.116947472745292, 'pattern_counts': {'":': 8, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '127385', 'license': 'MIT', 'n_tokens_mistral': 2566, 'n_tokens_neox': 2391, 'n_words': 521, 'path': 'Chapter 13/Chapter_13.ipynb', 'repo_name': 'bpbpublications/Applied-Machine-Learning-Solutions-with-Python', 'size': '6171'}