<jupyter_start><jupyter_text>Broadcasting [matrixmultiplication.xyz](matrixmultiplication.xyz) When operating on two arrays/tensors, Numpy/PyTorch compares their shapes element-wise. It starts with the **trailing dimensions**, and works its way forward. Two dimensions are **compatible** when    - they are equal, or,    - one of them is 1, in which case that dimension is broadcasted to make it the same sizeArrays do not need to have the same number of dimensions. For example, if you have a `256*256*3` array of RGB values, and you want to scale each color in the image by a different value, you can multiply the image by a one-dimensional array with 3 values. Lining up the sizes of the trailing axes of these arrays according to the broadcast rules, shows that they are compatible:       Image  (3d array): 256 x 256 x 3              Scale  (1d array):             3              Result (3d array): 256 x 256 x 3              The [numpy documentation](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.htmlgeneral-broadcasting-rules) includes several examples of what dimensions can and can not be broadcast together.  Einstein summation<jupyter_code>def matmul4(a,b):
    return torch.einsum('ik,kj->ij',a,b)
%timeit -n 10 t1 = matmul4(a,b)
%timeit -n 10 a.matmul(b)<jupyter_output>The slowest run took 5.77 times longer than the fastest. This could mean that an intermediate result is being cached.
115 µs ± 110 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>RELU<jupyter_code>def normalize(x,m,s):
    return (x-m)/s
mean, std = x_train.mean(),x_train.std()
mean, std
x_train = normalize(x_train,mean,std)
x_valid = normalize(x_valid,mean,std)
mean, std = x_train.mean(),x_train.std()
mean, std
nh = 50

# kaiming init
w1 = torch.randn(m,nh)/math.sqrt(2/m)
b1 = torch.randn(nh)
w2 = torch.randn(nh,1)/math.sqrt(2/m)
b2 = torch.rand(1)

def relu(x):
    return x.clamp_min(0.)-.5<jupyter_output><empty_output><jupyter_text>if use normal relu without -.5 your mean will be .5 and not 0. seems legit<jupyter_code>def lin(x,w,b):
    return x@w+b
relu(lin(x_valid,w,b))
w1 = torch.zeros(m,nh)
torch.nn.modules.conv._ConvNd.reset_parameters??<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1057866184448463, 'fraction_numerical': 0.029837251356238697, 'mean_word_length': 4.207058823529412, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '584437', 'license': 'Apache-2.0', 'n_tokens_mistral': 792, 'n_tokens_neox': 734, 'n_words': 259, 'path': 'own_notes/notes on fastai coursev3.8 .ipynb', 'repo_name': 'takotab/course-v3', 'size': '2212'}