<jupyter_start><jupyter_text>Scientific ApplicationsFollowing are important components of a typical scientific application:* Numerical computations, matrix operations* Data visualization through 2D and 3D graphs* GUI with event-driven architecture* Reports and output in print-friendly format* Access to databases and datastores* Access to Internet servicesWhile all of the above are important components, the speed of numerical computations determine the success of a scientific application. While Python is quite strong in the rest of the requirements, it is severely deficient in speed. There are many reasons for this:* Python is interpreted* Python is a weakly typed language. Hence, it is acceptable to dynamically change the data type of an object at run-time. This makes determining Run Time Type Information (RTTI) necessary to perform operations on the data, thereby slowing the performance Improving Performance of Python ProgramsIt is possible to improve performance of a program by choosing an efficient algorithm. But improvement in performance that can be achieved by writing code in a faster programming languae than Python, such as, C/C++ can enhance performance even of the best algorithm. Therefore the best approach to improving performance is by writing code in C, in such a way that it can be called in Python. The typical workflow for this is as follows:* Write code in C/C++* Include required Python header files and write required wrapper code so that it can be called in Python* Compile the C/C++ code to a importable shared library using a C/C++ compiler such as **gcc*** Import the module in Python and test the code CythonThe **Cython** project takes a slightly different approach but essentially does the same. Cython is a programming language based on Python but with additional syntax allowing for optional static type declarations, essentially making it a strongly typed language. The typical workflow is as follows:* Code is written in Cython, which is a superset of Python, with additional static type declarations which the programmer thinks will speed up performance. Files have the extension **.pyx*** Cython program is compiled to a C program by the Cython compiler. Files have the same first name as the .pyx file from which they are derived and an extension **.c*** The resulting C program is compiled to a shared importable Python library using **gcc** C compiler. The resulting file has the same first name as the original .pyx file with an extension **.so** (GNU/Linux), **.pyd** (Windows).* The resulting Python is imported into Python and testedExcept for the first step, the workflow is the same as before. But there is a definite advantage in writing code in Python itself. As a result, any Python program is also a Cython program and is a potential candidate for performance enhancement. As anadded advantage, Cython makes it easy to integrate Python with external C libraries. Cython RequirementsTo use Cython, the following are required:* Cython module* Python development package containing header files required to compile C/C++ code to be compatible with Python* GNU C compiler **gcc** or GNU C++ compiler **g++*** Depending on how you intend to use Cython, one of the following may be required:    * Python **distutils** module - The steps of compiling Cython code to C and then compiling it to importable Python module can be automated by writing a distutils **setup.py** file    * Python **pyximport** module - Automation can be further be simplified using pyximport module Building a Cython Module using distutilsFollowing are the typical steps to use **distutils**:* Code is written in Cython, which is a superset of Python. The programmer may choose to include static type declarations. Let us assume that a Cython file **hello.pyx** contains a simple **say_hello()** function* A distutils setup.py file is written which contains the name to be used for the importable Python module and the Cython file where code resides* The Python importable module is builtHere is the Cython file **hello.pyx** containing the famous funtion that greets the World:    def say_hello():        print "Hello world!"The corresponding **setup.py** file that uses `distutils` is given below:    from distutils.core import setup    from Cython.Build import cythonize    setup(      ext_modules = cythonize("helloworld.pyx")    )The `setup.py` file imports the required **`setup`** function and **`Extension`** class from the submodules `distutils.core` and `distutils.extension`, respectively. It imports `build_ext` class from `Cython.Distutils` submodule.The name of the Cython source file `hello.pyx` and the name to be assigned to the Python importable module, namely, `hello` are also specified.The hello module is built using the following command:    $ python setup.py build_ext --inplaceThis produces the **hello.so** file which can be imported into Python and used with as follows:    import hello    hello.say_hello()This produces the output:    Hello, World! Building a Cython Module from the Command LineRun the Cython compiler to convert the `.pyx` file to generate a `.c` file    $ cython -a hello.pyxThe `-a` switch, which is optional, generates the `hello.html` HTML file which can be opened in a browser. This contains the code with the lines highlighted in different shades of yellow. Lines in bright yellow highlighting require optimisation while lines with white background translate into direct C code. This generates the C file `hello.c`.The generated C code `hello.c` must be compiled to an importable Python module `hello.so` (`hello.pyd` in Windows) using the C compiler:    gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing \      -I/usr/include/python2.7 -o hello.so hello.cIt is important to tell where the `Python.h` header file is located using the `-I` switch. The generated shared object file `hello.so` can then be imported into a Python program using the `import hello` statement and used just as you would, any other module.    import hello        hello.say_hello('World')Running this Python program generates the following output on the console:    Hello, World! Building a Cython Module using pyximportUsing `pyximport` module makes it unnecessary to write a `setup.py` file. The typical `import` statements while using `pyximport` are as follows:    import pyximport    pyximport.install()    import hello    print hello.say_hello('World')The `import hello` statement actually carries out the task of compiling the `.pyx` file to a `.c` file. This C file is then compiled to a `.so` (or a `.pyd`) file by the `import hello` statement. In fact, everythin happens transparently and you will not see the `.c` and `.so` files at all. If the first two lines where you import pyximport and the next line where we call the `install()` function, you would not realise that several things are happening behind the scenes and you would think you were running an ordinary Python script.<jupyter_code>import pyximport
pyximport.install()

import hello
hello.say_hello()<jupyter_output><empty_output><jupyter_text>Cython ExampleLet us demonstrate how Cython can improve performance of Python code by starting with a simple pure Python function and bring in speed improvements in phases. Pure Python CodeLet us first write a pure Python script file `quad.py`, containing two functions. The first function `def f(a, b, N):`The **IPython** shell provides a **magic** function `%timeit` to measure the time of execution of selected functions. It executes the specified function a certain number of times and reports the best time. This feature of **IPython** is used as shown on the last line above.<jupyter_code># file: quad.py - Pure Python code
from math import sin

def f(x):
  return sin(x**2)

def integrate(a, b, N):
  s = 0
  dx = float(b - a) / N
  for i in range(N+1):
    s += f(a + i*dx)
  return s * dx

%timeit integrate(0, 1, 100000)<jupyter_output>10 loops, best of 3: 50.6 ms per loop<jupyter_text>Cythonize the Python Script (No Changes to Python Code)As afirst attempt, let us simply Cythonize this script without attempting any optimizations. Let us copy this Python file  **`quad.py`** as it is into **`quad0.pyx`** file and using **`pyximport`**, import and time the function and compare the execution speeds:<jupyter_code>import pyximport
pyximport.install()

import quad0
%timeit quad0.integrate(0, 1, 100000)<jupyter_output>10 loops, best of 3: 26.8 ms per loop<jupyter_text>The execution time reduced from 50.6 ms to 26.8 ms, a reduction of 47% compared to the pure Python code. Static Type Declaration of Function Parameters and Function Local VariablesPerformance can be further improved if the data types of the variables can be specified. Here is the **`quad0.pyx`** modified and named as **`quad1.pyx`**      file: quad1.pyx - With type declaration    from math import sin    def f(double x):      return sin(x*x)    def integrate(double a, double b, int N):      cdef int i      cdef double s, dx      s = 0.0      dx = float(b - a) / N      for i in range(N):        s += f(a + i*dx)      return s * dx The only changes to the pure Python code that we have made are the static data type declarations for the function parameters and local variables inside the function `integrate()`. To compile and execute this script, we will again use `pyximport`<jupyter_code>import pyximport
pyximport.install()

import quad1
%timeit quad1.integrate(0, 1, 100000)<jupyter_output>100 loops, best of 3: 10.2 ms per loop<jupyter_text>The execution time is now reduced to 10.2 ms, a reduction of 79% compared to the pure Python script. Identify What to Speed UpCython can help us in identifying which part of the script needs improvement. To do this, run the `cython` compiler with the `-a` switch on the command line as follows:    cython -a quad1.pyxThis generates an HTML file `quad1.html` which you can open in a browser. Lines marked with yellow background are potential candidates where speed improvements can be achieved. Thus, the return type of the function `f()` can help speed up the program.<jupyter_code>from IPython.display import HTML
HTML(filename='quad1.html')<jupyter_output><empty_output><jupyter_text>Static Type Declaration of Function Return ValueThe modified script is now named `quad2.pyx` and is shown below:      file: quad2.pyx - Type declaration and function return value        from math import sin    cpdef double f(double x):      return sin(x*x)    def integrate(double a, double b, int N):      cdef int i      cdef double s, dx      s = 0.0      dx = float(b - a) / N      for i in range(N):        s += f(a + i*dx)      return s * dx Let us now execute this script and see what improvement it can make to the code:<jupyter_code>import pyximport
pyximport.install()

import quad2
%timeit quad1.integrate(0, 1, 100000)<jupyter_output>100 loops, best of 3: 9.86 ms per loop<jupyter_text>The execution time is now down to 9.86 ms, a reduction of 80.5% compared to the pure Python script. Import Functions from C Standard LibraryAnother possibility is to call the `sin` function from the C `math` library directly. Here is the modified `quad3.pyx` file with the changes made to the `import` statement on the first line:      file: quad3.pyx - Import function from C standard library    from libc.math cimport sin    cpdef double f(double x):      return sin(x*x)    def integrate(double a, double b, int N):      cdef int i      cdef double s, dx      s = 0.0      dx = float(b - a) / N      for i in range(N):        s += f(a + i*dx)      return s * dx Let us test this modified script and see what happens:<jupyter_code>import pyximport
pyximport.install()

import quad3
%timeit quad1.integrate(0, 1, 100000)<jupyter_output>100 loops, best of 3: 9.84 ms per loop<jupyter_text>The execution time does not show any significant improvement. It can be seen that speed can be increased dramatically by using Cython by statically declaring the types of function parameters, function local variables, function return values and calling functions directly from the C library.  Cython Magic Functions in IPythonIPython provices a cythonmagic extension that contains a number of magic functions to simplify working with Cython. This extension is loaded with the **`%load_ext`** magic as follows:    %load_ext cythonmagicOnce this extension is loaded, several Cython related magic become available.<jupyter_code>%load_ext cythonmagic
%%cython_pyximport foo
def f(x):
    return 4.0 * x
print f(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06355661881977671, 'fraction_numerical': 0.01028708133971292, 'mean_word_length': 4.3939784946236555, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '337555', 'license': 'CC0-1.0', 'n_tokens_mistral': 3313, 'n_tokens_neox': 3159, 'n_words': 1814, 'path': 'pybelgaum/03_Cython.ipynb', 'repo_name': 'satish-annigeri/Notebooks', 'size': '12540'}