<jupyter_start><jupyter_text>CS207 Project Group 11 - "The Differentiators" Milestone 2*****  IntroductionDerivatives are ubiquitous in many fields such as engineering design optimization, fluid dynamics and machine learning.There are in general three ways to calculate the derivatives: automatic differentiation, numeric differentiation, andsymbolic differentiation. Automatic Differentiation (AD) brings a family of techniques that can calculate the partialderivatives of any function at any point efficiently and accurately. Unlike numeric differentiation, AD does not havethe problem of floating point precision errors, since it calculates the derivative of a simple function, and keeps track ofthese derivatives, and there is no need of step sizes. Compared to symbolic differentiation, AD is not as memory intense,and can be much faster in terms of the calculation. Therefore, AD is an important way to calculate derivatives inpractice.The software that we design calculates the derivatives given the user’s input using the forward mode of automatic differentiation,and provides the user with an easy way to solve their optimization problem using derivatives.  BackgroundAt the core of Automatic Differentiation is the principle that functions implemented as computer code can be broken down into elementary functions, ranging from arithmetic operations (e.g. addition, subtraction etc.) and other functions (e.g. power, exponential, sin etc.). Hence, any differentiable function can be interpreted as a composition of different functions. For example, given a function, $f = sin^2(2x)$, it can be rewritten as:$$ f = \phi_1(\phi_2(\phi_3(x))) $$ where $$ \phi_1(z) = z^2,   \phi_2(y) = sin(y) \text{ and } \phi_3(x) = 2x$$In the forward mode, the chain rule can then be applied successively to each elementary component function to obtain the derivative of the function. Using the same example above, let $c$ be a real number:$$ f'(c) =  \phi_3'(\phi_2(\phi_1(c))) \cdot \phi_2'(\phi_1(c)) \cdot \phi_1'(c)$$Based on the example above, the derivative, $f'(c)$, can be evaluated based on the following function-derivative pairs at each stage of computing the function:$$(\phi_1(c), \phi_1'(c))$$$$(\phi_2(\phi_1(c)), (\phi_2'(\phi_1(c)) \cdot \phi_1'(c)))$$$$(\phi_3(\phi_2(\phi_1(c))), \phi_3'(\phi_2(\phi_1(c)) \cdot \phi_2'(\phi_1(c)) \cdot \phi_1'(c))$$Effectively, the forward mode computes the Jacobian-vector product, $Jp$. This decomposition can be represented via a computational graph structure of calculations, requiring initial values to be set for $x_1$, and $x'_1$:$$x_1 \rightarrow^{\phi_3(x)} x_2 \rightarrow^{\phi_2(x)} x_3 \rightarrow^{\phi_1(x)} y $$where $$ \phi_1(x) = x^2,   \phi_2(x) = sin(x) \text{ and } \phi_3(x) = 2x$$At each stage of the function, the derivative of the function with respect to its argument is calculated. The exact values of the function and its derivative are used for the following function-derivative pair of values. An example of the computational trace for the equation $f = sin^2(2x)$ would look like this, for $x = \dfrac{\pi}{6}$. | Trace    | Elementary Operation &nbsp;&nbsp;&nbsp;| Derivative &nbsp;&nbsp;&nbsp; | $\left(f\left(a\right),  f^{\prime}\left(a\right)\right)$&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|| :------: | :----------------------:               | :------------------------------: | :------------------------------: || $x_{1}$  | $\dfrac{\pi}{6}$                       | $1$                | $\left(\dfrac{\pi}{6}, 1\right)$ || $x_{2}$  | $2x_{1}$                               | $2\dot{x}_{1}$     | $\left(\dfrac{\pi}{3}, 2\right)$ || $x_{3}$  | $\sin(x_{2})$               | $\cos\left(x_{2}\right)\dot{x}_{2}$            | $\left(\dfrac{\sqrt{3}}{2}, 1\right)$ || $x_{4}$  | $x_{3}^{2}$                            | $2x_{3}\dot{x}_{3}$                   | $\left(\dfrac{3}{4}, \sqrt{3}\right)$ |By evaluating the derivative at each step of the chain rule, we eventually obtain the value of the derivative $f'(x) = \sqrt{3}$ at $x = \dfrac{\pi}{6}$, as second entry of the final tuple in the table.While the above illustrates the forward mode of AD (the focus of our package), AD also has a reverse mode. Without using chain rule, it first does a forward pass to store the partial derivatives, before undertaking a reverse pass, which starts from the final function to be differentiated, $y$. After fixing the derivative of the final function, it then computes the derivative of each component function with respect to its parent function recursively (using chain rule) until the derivative of the function with respect to the basic-level argument (e.g. $x_1$) can be calculated. In terms of efficiency, the forward mode is more efficient when the number of functions to evaluate is much greater than the number of inputs, whereas the reverse mode, which computes the Jacobian-transpose-vector-product is more efficient when the number of inputs is much greater than the number of functions.  How to use `AutoDiff`At this milestone, the recommended method to access the package is to download or clone the package's Github repo (https://github.com/the-differentiators/cs207-FinalProject.git). The user must ensure that the package's requirements (`numpy`) are installed, or install them manually.By the final submission, the package will be available on PyPI and the user will be able to install the package in the standard way using `pip`. We will also provide a `yml` file which will be used by the user to create the appropriate environment and ensure that dependencies like `numpy` have been installed. Then, to use the package, the user will only need to import our package which will implicitly import any other packages used by `AutoDiff`.The following steps will walk the user through a demo of how to install and use the `AutoDiff` package:  Importing `AutoDiff` and requirementsThe following code is first used to change the path directory of this document to the same folder as that of the package<jupyter_code># Code to change directory to src folder
import os
path = os.getcwd().replace('docs','src')
os.chdir(path)<jupyter_output><empty_output><jupyter_text>Once the package's Github repo has been downloaded, the `AutoDiff` package can be imported with the following code by a Python file, with its working directory configured to the same directory as `AutoDiff.py`:<jupyter_code>import AutoDiff as AD<jupyter_output><empty_output><jupyter_text>For the purposes of this demo, we will import `numpy`, which is a requirement for the *AutoDiff* package, as well as the `Ad_Var` class from the `AutoDiff` package<jupyter_code>import numpy as np
from AutoDiff import Ad_Var<jupyter_output><empty_output><jupyter_text>Using `AutoDiff` to compute derivative of a scalar function Below, we have included a basic demo for a scalar function, given a single input. The function used in the demo is  $f = sin^2(2x)$, which was used for illustration in the *Background* section earlier. Our objective is to use the the `Ad_Var` class to compute the value of the derivative for this function automatically, unlike the manual computational trace drawn out earlier. First, we create an instance of the `Ad_Var` object, with the value of $x = \dfrac{\pi}{6}$ assigned to the input variable, `val`.<jupyter_code>a = np.pi / 6

x = Ad_Var(a)<jupyter_output><empty_output><jupyter_text>The user should note that the `AutoDiff` package assumes that for a single input, the object being initialised will have a derivative value of 1 (stored as a Class attribute `self._ders`). Next, we create `f`, which represents the full function. The `Ad_Var` object from the previous code can be used with dunder functions and additional functions within `Ad_Var` class to construct the full function being evaluated.<jupyter_code>f = (Ad_Var.sin(2*x))**2<jupyter_output><empty_output><jupyter_text>As the functions are applied to the original `Ad_Var` object `x`, the `_val` and `_ders` attributes of the object are being updated with new values. The object `f`, representing the full function, will have its `_val` and `_ders` attributes containing the actual function and derivative values respectively.To note: the user does not have the ability to manually set function and derivative values outside of instance initialization. Instead, the `_val` and `_ders` attributes will be made pseudoprivate.The associated function value and derivative(s) of any `Ad_Var` instance may be retrieved through the `get_val` and `get_ders` functions as shown below:<jupyter_code>print(f.get_val(), f.get_ders())<jupyter_output>0.7499999999999999 1.7320508075688776<jupyter_text>This almost returns us the values that we would have computed manually, except for the rounding error from floating point operations in Python. To verify this, we run the following assert statements on the function and derivative values computed using `numpy`, using $f = sin^2(2x)$ and $f' = 4sin(2x)cos(2x)$<jupyter_code>val = np.pi / 6

assert(f.get_val()== (np.sin(2*val)**2))
assert(f.get_ders()== (4*np.sin(2*val)*np.cos(2*val)))<jupyter_output><empty_output><jupyter_text>Software Organization  Directory structureOur intended directory structure is as follows:```cs207-FinalProject/                   README.md                   requirements.txt                    docs/                        milestone1.pdf                        milestone2.ipynb                   src/                                           AutoDiff.py                   test/                        test_autodiff.py                       ```                  ModulesThe primary module will be a single `AutoDiff.py` file. Contained within will be the definition for an `Ad_Var` class. Instances of this class, through interaction with other `Ad_Var` objects, will be able to compute the value of a function as well as the value of that function's derivative with respect to any input variable. At present, we envision that this module will be powerful enough to handle forward differentiation of any function comprised of the following elementary functions:* Fundamental arithmetic operators (addition, subtraction, multiplication, and division)* Logarithm (of any base)* Negation* Exponentiation ($e^x$ for an `Ad_Var` instance $x$)* Power and root functions ($x^n$ for some real $n$)* Trigonometric functions ($\sin(x)$, $\cos(x)$, $\tan(x)$)* Inverse trigonometric functions ($\arcsin(x)$, $\arccos(x)$, $\arctan(x)$)Depending on our eventual choice for the "additional" feature of this project, or future design decisions, theremay be additional modules added in the future that supplement or subdivide the functionality of `AutoDiff.py`.Each instance of the `Ad_Var` class in the `AutoDiff` package represents the definition of a set of variables at a particular evaluation point. Through manipulations of these instances (either through fundamental arithmetic operations or built-in methods representing additional elementary functions described earlier), a user has the capability of representing any continuous differentiable function, be it scalar or vector. This was shown earlier via a code demo. Testing and CoverageIn the `test` folder, there will be a separate Python module `test_autodiff.py`, which will be the test-suite for `AutoDiff.py`. The test-suite will contain tests for the methods in the `Ad_Var` class, to ensure that the elementary functions return the desired output. Tests are run using pytest. The tests are linked to Travis CI and CodeCov, which will manage continuous integration and code coverage respectively. Installation and distribution of packageAt this milestone, the recommended method to access the package is to download or clone the package's Github repo (https://github.com/the-differentiators/cs207-FinalProject.git). The user must ensure that the package's requirements (`numpy`) are installed, or install them manually.By the final submission, we intend to distribute the package via PyPI. There will be no additional packaging framework included; we believe the scope of this project can be contained within a relatively simple directory structure with few functional python files and should not require additional overhead for users to install and use.The package will be available on PyPI and the user will be able to install the package in the standard way using `pip`. We will also provide a `yml` file which will be used by the user to create the appropriate environment and ensure that dependencies like `numpy` have been installed. Then, to use the package, the user will only need to import our package which will implicitly import any other packages used by `AutoDiff`.  Implementation Details Core Data Structures* `numpy` arrays: 1-D `numpy` arrays will be used to keep the gradient vectors as the entire trace is evaluated. `numpy`provides vectorized operations which will make the overloading of elementary functions much more efficient formultivariate functions. If a vector function is provided, 2-D `numpy` arrays will be used to hold the Jacobian matrix. Class Implementation* The `Ad_Var`  class will represent the variables that are used in the Automatic Differentiation process. In the case of a single input, the instance should be initialized with, `val`, a scalar value of that variable to be evaluated on when calculating both the function and derivative values (as shown in the demo above)* In the case of multiple inputs, each input will be initialized as an `Ad_Var` object, with inputs `val`, a scalar value of that variable and `ders`, a `numpy` array representing the derivative of the input with regards to the other variables. An example is shown below:<jupyter_code>x1 = Ad_Var(1, np.array([1, 0, 0]))
x2 = Ad_Var(2, np.array([0, 1, 0]))
x3 = Ad_Var(3, np.array([0, 0, 1]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09234282452008849, 'fraction_numerical': 0.011774780560907729, 'mean_word_length': 4.429678419217358, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '500519', 'license': 'MIT', 'n_tokens_mistral': 3681, 'n_tokens_neox': 3491, 'n_words': 1792, 'path': 'docs/milestone2.ipynb', 'repo_name': 'the-differentiators/cs207-FinalProject', 'size': '14013'}