<jupyter_start><jupyter_text>Homework 1 - Berkeley STAT 157Handout 1/22/2017, due 1/29/2017 by 4pm in Git by committing to your repository. Please ensure that you add the TA Git account to your repository.1. Write all code in the notebook.1. Write all text in the notebook. You can use MathJax to insert math or generic Markdown to insert figures (it's unlikely you'll need the latter). 1. **Execute** the notebook and **save** the results.1. To be safe, print the notebook as PDF and add it to the repository, too. Your repository should contain two files: ``homework1.ipynb`` and ``homework1.pdf``. The TA will return the corrected and annotated homework back to you via Git (please give `rythei` access to your repository).<jupyter_code>!pip install -U mxnet-cu101mkl==1.6.0  # updating mxnet to at least v1.6
from mxnet import ndarray as nd
import mxnet as mx
import numpy as np
import time<jupyter_output><empty_output><jupyter_text>1. Speedtest for vectorizationYour goal is to measure the speed of linear algebra operations for different levels of vectorization. You need to use `wait_to_read()` on the output to ensure that the result is computed completely, since NDArray uses asynchronous computation. Please see http://beta.mxnet.io/api/ndarray/_autogen/mxnet.ndarray.NDArray.wait_to_read.html for details. 1. Construct two matrices $A$ and $B$ with Gaussian random entries of size $4096 \times 4096$. 1. Compute $C = A B$ using matrix-matrix operations and report the time. 1. Compute $C = A B$, treating $A$ as a matrix but computing the result for each column of $B$ one at a time. Report the time.1. Compute $C = A B$, treating $A$ and $B$ as collections of vectors. Report the time.1. Bonus question - what changes if you execute this on a GPU?  practice<jupyter_code>#A = nd.random.normal(shape=(3,3))
A = nd.random.normal(shape=(3,3))
#A = nd.random.normal(0, 1, shape=(4096,4096))
#B = nd.random.normal(shape=(4096,4096))
B = nd.random.normal(shape=(3,3))
print("A[0:3,0:3]= ", A[0:3,0:3])
tic = time.time()
#C = A * B
C = nd.dot(A, B)
C.wait_to_read()
print("Matrix by matrix: " + str(time.time() - tic) + " seconds")
print("C[0:3,0:3]= ", C[0:3,0:3])
print("A[0:3,0:3]= ", A[0:3,0:3])
tic = time.time()
D = A * B
#C = nd.dot(A, B)
#D = np.matmul(A,B)
#D.wait_to_read()
#print("Matrix by matrix: " + str(time.time() - tic) + " seconds")
print("D[0:2,0:2]= ", D )
print("A[0,0]= ", A[0:2,0:2])
tic = time.time()
for i in range A.size(1,0)
X = np.arange(4).reshape(2, 2)
X
X.shape
X.size
type(X)
X = nd.arange(4).reshape(2, 2)
X
type(X)
#X=np.array([2,2],[3,3])
#u = np.array([[3, -4, 3], [3, 5, 5]] )
#u = np.array([[1,2],[5,2]])
u = np.array([[[1,1],[2,2]],[[5,5],[2,6]]])
print(u.shape)
print(u)
print(type(u))
X = nd.arange(8).reshape(2, 2, 2)
print(X.shape)
print(X)
print(type(X))
Y = nd.arange(8).reshape(2,2,2)
Y
Z = nd.dot(X,Y)
Z<jupyter_output><empty_output><jupyter_text>nd.array practice<jupyter_code>A = nd.array([[2, 4], [3, 1]])
B = nd.array([[3, 1], [2, 5]])
AA = A.asnumpy()
BB = B.asnumpy()
AA
type(AA)
A
A
B
nd.dot(A,B)
A*B
#C= np.matmul(A,B)
C= np.matmul(AA,BB)
type(C)
C.shape
#C.size
print(C)
import numpy
# Two matrices are initialized by value
x = numpy.array([[1, 2], [4, 5]])
y = numpy.array([[7, 8], [9, 10]])
print(x, y)
#  add()is used to add matrices
print ("Addition of two matrices: ")
print (numpy.add(x,y))
# subtract()is used to subtract matrices
print ("Subtraction of two matrices : ")
print (numpy.subtract(x,y))
# divide()is used to divide matrices
print ("Matrix Division : ")
print (numpy.divide(x,y))
print ("Multiplication of two matrices: ")
print (numpy.multiply(x,y))
print ("The product of two matrices : ")
print (numpy.dot(x,y))
print ("square root is : ")
print (numpy.sqrt(x))
print ("The summation of elements : ")
print (numpy.sum(y))
print ("The column wise summation  : ")
print (numpy.sum(y,axis=0))
print ("The row wise summation: ")
print (numpy.sum(y,axis=1))
# using "T" to transpose the matrix
print ("Matrix transposition : ")
print (x.T)
np.matmul(x,y)
np.dot(x,y)
x*y
np.multiply(x,y)<jupyter_output><empty_output><jupyter_text>solution<jupyter_code>A = nd.random.normal(shape=(4096,4096))
B = nd.random.normal(shape=(4096,4096))
A.shape
tic = time.time()
C = nd.dot(A,B)
C.wait_to_read()
print("Matrix by matrix: " + str(time.time() - tic) + " seconds")
tic = time.time()
C = nd.dot(A,B)
C = nd.empty((4096, 4096))
tic = time.time()
for i in range(4096):
    C[:, i] = nd.dot(A, B[:, i])
C.wait_to_read()
print("Matrix by vector: " + str(time.time() - tic) + " seconds")
C = nd.empty((4096, 4096))
tic = time.time()
for i in range(4096):
    for j in range(4096):
        C[i,j] = nd.dot(A[i, :], B[:, j])
C.wait_to_read()
print("Vector by vector: " + str(time.time() - tic) + " seconds")<jupyter_output><empty_output><jupyter_text>2. Semidefinite MatricesAssume that $A \in \mathbb{R}^{m \times n}$ is an arbitrary matrix and that $D \in \mathbb{R}^{n \times n}$ is a diagonal matrix with nonnegative entries. 1. Prove that $B = A D A^\top$ is a positive semidefinite matrix. 1. When would it be useful to work with $B$ and when is it better to use $A$ and $D$?   3. MXNet on GPUs1. Install GPU drivers (if needed)1. Install MXNet on a GPU instance1. Display `!nvidia-smi`1. Create a $2 \times 2$ matrix on the GPU and print it. See http://d2l.ai/chapter_deep-learning-computation/use-gpu.html for details.<jupyter_code>!nvidia-smi
x = nd.ones((2, 2), ctx=mx.gpu())
x
AA = nd.random.normal(shape=(4096,4096),ctx=mx.gpu())
BB = nd.random.normal(shape=(4096,4096),ctx=mx.gpu())
tic = time.time()
C = nd.dot(AA,BB)
C.wait_to_read()
print("Matrix by matrix: " + str(time.time() - tic) + " seconds")
C = nd.empty((4096, 4096))
tic = time.time()
for i in range(4096):
    C[:, i] = nd.dot(AA, BB[:, i])
C.wait_to_read()
print("Matrix by vector: " + str(time.time() - tic) + " seconds")
C = nd.empty((4096, 4096))
tic = time.time()
for i in range(4096):
    for j in range(4096):
        C[i,j] = nd.dot(A[i, :], B[:, j])
C.wait_to_read()
print("Vector by vector: " + str(time.time() - tic) + " seconds")<jupyter_output><empty_output><jupyter_text>4. NDArray and NumPy Your goal is to measure the speed penalty between MXNet Gluon and Python when converting data between both. We are going to do this as follows:1. Create two Gaussian random matrices $A, B$ of size $4096 \times 4096$ in NDArray. 1. Compute a vector $\mathbf{c} \in \mathbb{R}^{4096}$ where $c_i = \|A B_{i\cdot}\|^2$ where $\mathbf{c}$ is a **NumPy** vector.To see the difference in speed due to Python perform the following two experiments and measure the time:1. Compute $\|A B_{i\cdot}\|^2$ one at a time and assign its outcome to $\mathbf{c}_i$ directly.1. Use an intermediate storage vector $\mathbf{d}$ in NDArray for assignments and copy to NumPy at the end.<jupyter_code>A = nd.random.normal(shape=(4096, 4096),ctx=mx.gpu())
B = nd.random.normal(shape=(4096, 4096),ctx=mx.gpu())
c = np.empty(4096)
tic = time.time()
for i in range(4096):
    c[i] = (nd.norm(nd.dot(A, B[:, i])).asscalar())**2
print("One at a time: " + str(time.time() - tic) + " seconds")
d = nd.empty(4096)
tic = time.time()
for i in range(4096):
    d[i] = nd.norm(nd.dot(A, B[:, i]))**2
c = d.asnumpy()
print("Convert at end: " + str(time.time() - tic) + " seconds")<jupyter_output>Convert at end: 4.857295989990234 seconds<jupyter_text>5. Memory efficient computationWe want to compute $C \leftarrow A \cdot B + C$, where $A, B$ and $C$ are all matrices. Implement this in the most memory efficient manner. Pay attention to the following two things:1. Do not allocate new memory for the new value of $C$.1. Do not allocate new memory for intermediate results if possible.<jupyter_code>A = nd.random.normal(shape=(100,100),ctx=mx.gpu())
type(A)
A = nd.random.normal(shape=(100,100))
B = nd.random.normal(shape=(100,100))
C = nd.random.normal(shape=(100,100))
C += nd.dot(A,B)
type(C)<jupyter_output><empty_output><jupyter_text>6. Broadcast OperationsIn order to perform polynomial fitting we want to compute a design matrix $A$ with $$A_{ij} = x_i^j$$Our goal is to implement this **without a single for loop** entirely using vectorization and broadcast. Here $1 \leq j \leq 20$ and $x = \{-10, -9.9, \ldots 10\}$. Implement code that generates such a matrix.<jupyter_code>x = nd.arange(start=-10, stop=10.1, step=0.1).reshape((201, 1))
 x
j = nd.arange(start=1, stop=21, step=1.0).reshape((1, 20))
j.shape
print(x ** j)
Xij = x ** j
Xij.shape
print(Xij[0:5,:3])<jupyter_output>[[  -10.         100.       -1000.      ]
 [   -9.9         98.009995  -970.2989  ]
 [   -9.8         96.04      -941.1921  ]
 [   -9.7         94.09      -912.673   ]
 [   -9.6         92.16      -884.7361  ]]
<NDArray 5x3 @cpu(0)>
{'fraction_non_alphanumeric': 0.1722203137524333, 'fraction_numerical': 0.052444749799610675, 'mean_word_length': 4.296543359611886, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '599057', 'license': 'Apache-2.0', 'n_tokens_mistral': 3568, 'n_tokens_neox': 3324, 'n_words': 1034, 'path': 'homework1.ipynb', 'repo_name': 'blee8/berkeley-stat-157', 'size': '8733'}