<jupyter_start><jupyter_text>Making code fasterHello everyone! For this exercise, we will use Numba and Cython to make faster calculations.* **Numba** is a just-in-time compiler that generates fast machine code from our existing Python code.* **Cython** allows us to call C functions and declare C types on variables and class attributes, making high-performance code ([GitHub](https://github.com/cython/cython/wiki)).<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>**Numba**:When trying to enhance our code, some questions can pop out: * Will a specific module improve the performance of my code? * Is it going to be almost the same? Maybe worse?  **Numba** is a good option if our code involves a lot of math, "for loops," and NumPy.On the other hand, if we are working with Pandas, for example, the code won't be improved<jupyter_code>import pandas as pd
dsss = {'Semester week': [2, 3, 4, 5, 6], 'Lecture': ['Hardware', 'File handlng', 'Version control', 'Machine learning', 'Hardware']}

def pandas_function(our_dict):            # Function will not benefit from Numba jit
  df = pd.DataFrame.from_dict(our_dict)   # Numba doesn't know about pd.DataFrame
  return df.sort_values(by= ['Lecture'])  # Or this

print(pandas_function(dsss))
%%timeit 
pandas_function
!  pip install --upgrade numba # First, we need to upgrade numba to avoid some errors
from numba import jit, njit # jit: numba's dectorator, njit: alias for @jit(nopython=True)

pandas_numba = jit(pandas_function)
%%timeit
pandas_numba<jupyter_output>The slowest run took 37.09 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 5: 31.4 ns per loop<jupyter_text>As we see, the performance is almost the same.Let's check now with the next function:<jupyter_code>def movingAverage(x, window=7):
    
    result = np.zeros_like(x)

    for i in range(window, x.size):
        
        result[i] = x[i-window:i].mean()
        
    return result
x = np.linspace(0, 20, 10000)
y = np.sin(x)+np.random.randn(x.size)/4<jupyter_output><empty_output><jupyter_text>**1. To Do: use the ```nopython``` mode to run `movingAverage()` in machine code** Answer:<jupyter_code>movingAverageNumba = njit(movingAverage) #TODO<jupyter_output><empty_output><jupyter_text>**2. To Do: Calculate the moving average of y. Therefore use the `movingAverage() `function with and without the `nopython `mode. Compare both results using `%time`.** Answer:<jupyter_code>%time movingAverage(y) #TODO
%time movingAverageNumba(y) #TODO<jupyter_output>CPU times: user 147 ms, sys: 0 ns, total: 147 ms
Wall time: 150 ms<jupyter_text>**3. To Do: the first time using ``movingAverageNumba`` takes longer, why?** Answer: Because it has to compile the function **4. To Do: compare both functions using `%timeit`.**<jupyter_code>%timeit movingAverage(y) #TODO
%timeit movingAverageNumba(y) #TODO
# Plot:
plt.plot(x,y, label='y')
plt.plot(x, movingAverageNumba(y), label='movingAverageNumba')
plt.legend()<jupyter_output><empty_output><jupyter_text>Cython We can load `cythonmagic` to interactively work with Cython by using:<jupyter_code>%load_ext Cython
%%cython
def movingAverage_cython(x, window=7):
    result = np.zeros_like(x)
    for i in range(window, x.size):
        result[i] = x[i-window:i].mean()
        
    return result<jupyter_output>Error compiling Cython file:
------------------------------------------------------------
...
def movingAverage_cython(x, window=7):
    result = np.zeros_like(x)
            ^
------------------------------------------------------------

/root/.cache/ipython/cython/_cython_magic_8df45e96104368c5c2d0c532fa7e8d4b.pyx:2:13: undeclared name not builtin: np<jupyter_text>**5. To Do: fix the error, use ``timeit``, and plot the results.**<jupyter_code>%%cython
import numpy as np
def movingAverage_cython(x, window=7):
    result = np.zeros_like(x)
    for i in range(window, x.size):
        result[i] = x[i-window:i].mean()
    return result
%timeit movingAverage_cython(y)
plt.plot(x,y, label= 'y')
plt.plot(x, movingAverage_cython(y), label= 'movingAverage_cython')
plt.legend()<jupyter_output><empty_output><jupyter_text>ProfilingSometimes we wonder if there is a way to enhance the efficiency of our codes. A way to do it is by running a multiline process to find inefficient functions and operations: the so-called bottlenecks. To perform this multiline process, we need to install  ```line_profiler``` and load it using:<jupyter_code>! pip install line_profiler
%load_ext line_profiler
def sum_of_lists(N):
    total = 0
    for i in range(5):
        L = [j ^ (j >> i) for j in range(N)]
        total += sum(L)
    return total
def sum_of_lists_numpy(N):
    total = 0
    for i in range(5):
        L = np.array([j ^ (j >> i) for j in range(N)])
        total += np.sum(L)
    return total<jupyter_output><empty_output><jupyter_text>**6. To Do: perform a Line-By-Line profiling for both functions using the magic command ```%lprun```** Answer:<jupyter_code>%lprun -f sum_of_lists sum_of_lists(50) # TODO
%lprun -f sum_of_lists_numpy sum_of_lists_numpy(50) # TODO<jupyter_output><empty_output><jupyter_text>**7. To Do: Which function is faster? Why?** Answer: The function ``sum_of_lists`` is faster. Using ``np.sum()`` takes 20 times longer than ``sum()``.<jupyter_code>%timeit sum_of_lists(50)
%timeit sum_of_lists_numpy(50)<jupyter_output>10000 loops, best of 5: 80.7 µs per loop<jupyter_text>**8. Use Numba with ``sum_of_lists_numpy()`` to acelerate calculations.** Answer<jupyter_code>faster_function = njit(sum_of_lists_numpy) # TODO
%timeit faster_function(50)<jupyter_output>The slowest run took 242716.72 times longer than the fastest. This could mean that an intermediate result is being cached.
1 loop, best of 5: 1.08 µs per loop<jupyter_text>**9. If we run ``faster_function`` with the line-by-line profiler, it does not show the time consumed by each line of the code. Why?**<jupyter_code>%lprun -f faster_function faster_function(50)<jupyter_output><empty_output><jupyter_text>Answer: As Numba translates our code into fast machine code, it is complicated for a profiler to evaluate a specific line of the code.   Working with inefficient functions:<jupyter_code>from skimage import data, color
from skimage.transform import resize
from PIL import Image
imgs = np.uint8(data.lfw_subset()*255)
def res_skimage(imgs):
    new_size = (imgs[1].shape[0]//2, imgs[1].shape[1]//2)
    res_im = []
    for im in imgs:
        image_resized = resize(im, new_size, anti_aliasing=True) 
        res_im.append(image_resized)
    return np.asarray(res_im)
%time res_im_ski = res_skimage(imgs)
plt.subplot(1,2,1); plt.imshow(imgs[0], cmap='gray'); plt.title('Original')
plt.subplot(1,2,2); plt.imshow(res_im_ski[0], cmap='gray'); plt.title('Resized')<jupyter_output><empty_output><jupyter_text>**10. To Do: perform a Line-By-Line profiling, find the bottleneck and fix it using a faster function. Plot the results.** Answer:<jupyter_code>%lprun -f res_skimage res_skimage(imgs) # Line-By-Line profiling<jupyter_output><empty_output><jupyter_text>From the results: function ``resize()`` takes 99.7% of the time: approximately 428,2 ms out of 429,6ms. We can use Pillow to perform the same operation but faster:<jupyter_code>def res_pil(imgs):
    new_size = (imgs[1].shape[0]//2, imgs[1].shape[1]//2)
    res_im = []
    for im in imgs:
        im2 = Image.fromarray(im, 'L')
        image_resized = im2.resize(new_size)
        res_im.append(image_resized)
    return res_im
%time res_im_pil = res_pil(imgs)
%lprun -f res_pil res_pil(imgs)<jupyter_output><empty_output><jupyter_text>Now, the time consumed was distributed between reading each image and resizing them 54,9% (9,7 ms) and 42,6% (7,6 ms), respectively.<jupyter_code>plt.subplot(1,3,1); plt.imshow(imgs[0], cmap='gray'); plt.title('Original')
plt.subplot(1,3,2); plt.imshow(res_im_ski[0], cmap='gray'); plt.title('Resized')
plt.subplot(1,3,3); plt.imshow(res_im_pil[0], cmap='gray'); plt.title('Resized Pillow')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13303823927210132, 'fraction_numerical': 0.021763187015861305, 'mean_word_length': 4.772888573456352, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 68, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '93866', 'license': 'MIT', 'n_tokens_mistral': 2968, 'n_tokens_neox': 2826, 'n_words': 826, 'path': 'Exercise_10_solution.ipynb', 'repo_name': 'tuhinmallick/DataScience-Survival-Skils', 'size': '8133'}