<jupyter_start><jupyter_text>Introduction Since Newton invented calculus, differentiating a function has been essential to the advancement of humanity. Calculating the derivative of a function is crucial to finding the extrema for a function and determining zeros for a function, two operations that are central to optimization (1). Often, we can find the symbolic/analytical solution to the derivative of a function, however this has become increasingly complex and computationally expensive as our functions/equations have grown in size and complexity. Numerically solving differential equations forms a cornerstone of modern science and engineering and is intimately linked with machine learning; however this method suffers from rounding errors and numerical instability. Many of these issues can be solved using Automatic Differentiation (AD) because AD can calculate the exact derivative up to machine precision (2). The logic and processes behind AD enables it to be implemented using computer code, making it easily accessible for use by scientists and mathematicians. This python package will implement the forward mode of AD. ​  How to use the package  Installation Ideally, the user should have Anaconda installed (https://www.anaconda.com/download/).To install the package, a user will download or clone the repository onto their local machine. Then, the user can, but does not need to, create a virtual environment using virtualenv. If the user has Anaconda installed, then the user will already have virtualenv installed, but if they do not, the user can the user can run the following command in the terminal:    sudo easy_install virtualenvThen, to create a virtual environment, the user runs the following command in the terminal in the package directory:    virtualenv envTo activate the virtual environment, the user runs the following command in the terminal in the package directory:    source env/bin/activateTo deactivate the virtual environment, the user runs the following command in the terminal in the package directory:    deactivateWhether or not the user uses a virtual environment, they will need to make sure they have the necessary dependencies. In the package directory includes a requirements.txt file that lists the necessary dependencies. The user can easily install all the dependencies with the following command:    pip install -r requirements.txtAfter installing the necessary dependencies, the user can then start using the package in their code with an import statement such as    import autodiff as adTo run the tests for the package, the user can run the following command in terminal in the package folder:    pytest  Usage For example, suppose $f(x, y) = x^2 + 2xy + y^2$, and $x = 1, y = 2$.​$f(1, 2) = 9\\\frac{\partial f}{\partial x} = 2x + 2y \implies \frac{\partial f}{\partial x}|_{x = 1, y = 2} = 6 \\\frac{\partial f}{\partial y} = 2x + 2y \implies \frac{\partial f}{\partial y}|_{x = 1, y = 2} = 6 $​To solve this problem using our package, the user will run the following straightforward code.<jupyter_code>% cd ../
#Ignore this. Used to import autodiff in the docs folder.

import autodiff as ad

# Define x and y scalars with variable name and value. The actual python variable names do not matter.
x = ad.Scalar('x', 1)
y = ad.Scalar('y', 2)
# Express f in terms of x and y. User does not need to define any more scalar objects besides the basic ones x and y above. 
f = x ** 2 + 2 * x * y + y ** 2
# Get value of f(1, 2)
f.getValue()
# Get partial derivative with respect to x. Pass in variable names as String in list.
f.getGradient(['x'])
# Get partial derivative with respect to y
f.getGradient(['y'])
# Get both partial derivatives
f.getGradient(['x', 'y'])
f.getDeriv()<jupyter_output><empty_output><jupyter_text>With our package, users will be able to easily define functions that rely on variables and their derivatives. For example, consider Newton's method in finding roots: For a scalar function $f(x)$, we want to find the $x$ such that $f(x) = 0$. Newton's method states that we are able to find the root by updating from an intial `x_0` value with the update equation $x_{n} = x_{n-1} - \frac{f(x_{n-1})}{f^{\prime}(x_{n-1})} $ until we get an $x_n$ such that $f(x_n) \approx 0$. With our package, the method can be written as the following:<jupyter_code>def newton_method(fn, init_x = 0, threshold = 1e-10):
    # Initialize scalar with an init value
    x = ad.Scalar('x', init_x)
    y = fn(x)
    # While f(x) is greater than threshold, update x.
    while y.getValue() > threshold:
        x = x - y.getValue() / y.getGradient('x')
        y = fn(x)
    # Return x such that f(x) approximately equals 0.
    return x.getValue()<jupyter_output><empty_output><jupyter_text>Background​The following mathematical concepts and background are required for understanding automatic differentiation:​ 1. Differential calculus​Differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change.For example, given the function: \begin{align} f\left(x\right) &=  {x^{2}}      \end{align}  Increment x by h: \begin{align} f\left(x+h\right) &=  {(x+h)^{2}}      \end{align}  Apply the finite difference approximation to calculate the slope:  \begin{align} \frac{f\left(x+h\right) - f\left(x\right) }{h} \end{align} Simplify the equation:  \begin{align} &= \frac{x^{2}+2xh+h^{2}-x^{2} }{h}\\ &= \frac{2xh+h^{2}}{h}\\ &=2x+h \end{align}  Set $h\rightarrow 0$:   \begin{align} 2x +0 &= 2x  \end{align}  The derivative is then defined is:\begin{align} \lim_{h\to0} \frac{f\left(x+h\right) - f\left(x\right) }{h} \end{align}  2. Elementary functions and their derivatives|       Function $f(x)$                |       Derivative $f^{\prime}(x)$                || :-------------------:  | :------------------------------------------------------------------------------:  || ${c}$           | $0$         || ${x}$           | $1$         || ${x^{n}}$           | ${nx^{n-1}}$         || $\frac{1}{x}$     | $\frac{-1}{x^{2}}$     || $ln{x}$     | $\frac{1}{x}$     || $\sin(x)$           |   $\cos(x)$         || $\cos(x)$           |   $-\sin(x)$         || $\tan(x)$           |   $\dfrac{1}{\cos^2(x)}$         || $\exp(x)$           |   $\exp(x)$         || ${a^{x}}$           |   ${a^{x}\ln{a}}$         |  3. The chain rule$^{(1)}$For a function $h(u(t))$, the derivative of $h$ with respect to $t$ can be expressed as:$$\dfrac{\partial h}{\partial t} = \dfrac{\partial h}{\partial u}\dfrac{\partial u}{\partial t}.$$If the function is expressed as a combination of multiple variables that are expressed in terms of t, i.e. $h(u(t), v(t))$, the the derivative of $h$ with respect to $t$ can be expressed as:$$\frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}\frac{\partial u}{\partial t} + \frac{\partial h}{\partial v}\frac{\partial v}{\partial t}$$Note that we are only looking at scalar variables in this case, but this idea can be extended to vector variables as well.  For any $h\left(y\left(x\right)\right)$ where $y\in\mathbb{R}^{n}$ and $x\in\mathbb{R}^{m}$,    \begin{align}    \nabla_{x}h = \sum_{i=1}^{n}{\frac{\partial h}{\partial y_{i}}\nabla y_{i}\left(x\right)}.  \end{align} 4. The graph structure of calculations and forward accumulationForward accumulation is computing the derivative using the chain rule starting from the inner most derivative to the outer most derivative, where we assume the most basic variables have seed values. Using a graph helps visualize forward accumulation. For example,\begin{align} f\left(x,y\right) &= \frac{x}{y} +cos(x)sin(y)\\ x &= y = 1\end{align} ![](img/graph_eg.png)| Trace | Elementary Function | Current Value | Elementary Function Derivative | &nbsp; &nbsp; $\nabla_{x}$ Value &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;| &nbsp; &nbsp; $\nabla_{y}$ Value &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: || $w_{1}$ | $1$ | $1$ | $\dot{w_1}$ | $1$ | $0$ || $w_{2}$ | $1$ | $1$ | $\dot{w_2}$ | $0$ | $1$ || $w_{3}$ | $cos{(w_1})$ | $cos{(1)}$ | $-sin{(w_1)}\dot{w_1}$ | $-sin(1)$ | $0$ || $w_{4}$ | $sin{(w_2})$ | $sin{(1)}$ | $cos{(w_2)}\dot{w_2}$ | $0$ | $cos{(1)}$ || $w_{5}$ | $w_3\dot w_4$ | $sin{(1)}cos{(1)}$ | $w_4\dot{w_3} + w_3\dot{w_4}$ | $-sin^2{(1)}$ | $cos^2{(1)}$ || $w_{6}$ | $w_1 / w_2$ | $1$ | $\dot{w_1}/w_2 - w_1 \dot{w_2}/ w_2^2$ | $1$ | $-1$ || $w_{7}$ | $w_5 + w_6$ | $sin{(1)}cos{(1)} + 1$ | $\dot{w_5} + \dot{w_6}$ | $-sin^2{(1)} + 1$ | $cos^2{(1)}-1$ |  Software Organization The directory structure would look like that:        ``` autodiff\          autodiff\                    __init__.py                    functions.py                    scalar.py          tests\                    test_functions.py                    test_scalar.py           docs\                    milestone1.ipynb                    milestone2.ipynb          README.md          requirements.txt          setup.cfg          LICENSE                ```   **Basic modules and what they do?**This module aims to compute forward automatic differentiation. The module we implemented can efficiently compute the derivatives of a function of automatic differentiation of a scalar input. The *scalar.py* contains the objects which compute the scalar variables returning the value and the derivative. The dunder methods add, sub, mul, truediv, pow, iadd, isub, imul, idiv, ipow are implemented in this module. The *function.py* contains sine, cosine, power, and exponential functions. Both *scalar.py* and *function.py* return the value and the derivative.**Where do tests live? How are they run? How are they integrated?**We are using both `TravisCI` and `Coveralls` to test our module. Each test function exists in the tests folder, and it utilizes the pytest package.**How can someone install your package?**We are planning to release our package in `PyPI`, but have not done it yet. Therefore, our auto-differentiating software can be downloaded at https://github.com/cs207FinalProjectGroup/cs207-FinalProject.git.  Implementation Details  Current ImplementationWe have so far implemented the *Scalar* class, which represents scalar variables. To initialize a *Scalar* class object, the user will pass in a string that represents the variable (i.e. 'x', 'y', 'x1', etc.) and also the value of variable (the seed value). The *Scalar* class holds two attributes: 1) the value of the variable `_val` at the current step and 2) a dictionary `_deriv` containing the derivative or partial derivatives (keys will be the names of the variables (i.e *x* and *y*) and the values will be the derivative value with respect to each variable).<jupyter_code>import autodiff as ad

x = ad.Scalar('x',2);
print(x._val);
print(x._deriv);<jupyter_output>2.0
{'x': 1.0}<jupyter_text>Storing the partial derivatives with respect to each variable allows us to easily compute additional derivatives with respect to each variable when we are performing mathematical operations because we can update each partial derivative individually. When a *Scalar* object is initialized, by default `_deriv` will just be a dictionary with the only key being the string the user passes in with value 1. A user can access the value of a *Scalar* object using the *getValue()* method and access the derivative (or partial derivatives) for the object through the *getDeriv()* method. The user can also get the derivatives/partial derivatives as a numpy array with the *getGradient()* method, which takes in a list of strings, with each element representing the variable to take the derivative with respect to, as an argument. The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __itruediv__, __ipow__ (and the right equivalents for the ones that have one) have been overwritten so that they return a new *Scalar* object with an updated value and derivatives. Thus, the adding or substracting two Scalars or raising a Scalar to the power of another Scalar does not change the values or derivatives of the original Scalar objects. By overwriting these methods, we are implementing forward accumulation, as the orders of operation allows us to traverse the chainrule starting from the inside.<jupyter_code>#addition example
x,y = ad.Scalar('x', 2),ad.Scalar('y', 5);
val = x+y;
print(val.getValue()); #should be 7.0
print(val.getDeriv()['x']); #should be 1.0
print(val.getDeriv()['y']); #should be 1.0
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values
#pow example
x=ad.Scalar('x', 2);
val = x**2;
print(val.getValue()) #should be 4.0
print(val.getDeriv()['x']) #should be 4.0
#division example
x,y=ad.Scalar('x', 3),ad.Scalar('y', 2);
val = x/y;
print(val.getValue()) # should be 1.5
print(val.getDeriv()['x']); #should be 0.5
print(val.getDeriv()['y']); #should be -0.75
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values<jupyter_output>1.5
0.5
-0.75

3.0 , 2.0<jupyter_text>We have also implemented the functions **sin**, **cos**, **tan**, **power**, and **exp**. All of these functions can take in an `int`, `float`, or *Scalar* object. If only an `int`/`float` is provided, then a `float` is returned, but if *Scalar* object is provided, then a new *Scalar* object. No changes are made to the value or derivatives of the original *Scalar* object passed in. Within these functions, the *numpy* functions *sin*, *cos*, *tan* and *exp* are used to calculate the appropriate values. The trigonometric functions  **sin(x)**, **cos(x)**, and **tan(x)** each take in an `int`, `float`, or *Scalar* object *x* and apply the respective trigonometric function to the *x*. If a *x* is a *Scalar* object, the trigonometric function is applied to the *x._val* attribute and the derivative(s) is updated accordingly.<jupyter_code>import autodiff as ad
import numpy as np 

# Sine functions
x = ad.Scalar('x', 2.0) # float input 
val = ad.sin(x)
print(val.getValue()); #shoud be 0.90929742682
print(val.getDeriv()['x']); #should be -0.41614683654


# Cosine functions
x = ad.Scalar('x', 0) # integer input 
val = ad.cos(x)
print(val.getValue()); #should be 1.0
print(val.getDeriv()['x']); #should be 0.0

# tan function
x = ad.Scalar('x', 2.0); # float input 
val = ad.tan(x);
print(val.getValue()); #shoudl be -2.185039863261519
print(val.getDeriv()['x']); #should be 5.7743992040419174<jupyter_output>0.9092974268256817
-0.4161468365471424
1.0
-0.0
-2.185039863261519
5.774399204041917<jupyter_text>The **power(x1, x2)** function raises *x1* to the power of *x2*. *x1* and *x2* can be any combination of `ints`, `floats`, or *Scalar* objects. If only ints and floats are provided, then *power* will return a `float` with value *x1* raised to the power of *x2*. If at least one *Scalar* object is provided, then **power** works just like using the **__pow__** operator and returns a new *Scalar* object without changing any values in the original *Scalar* object(s).<jupyter_code>#power example
x = 5.0
p = 3.0
print (ad.power(x, p)) # should be 125.0<jupyter_output>125.0<jupyter_text>The **exp(x)** function raises *e* to the power of *x* , where *x* can be an `int`, `float`, or *Scalar* object. If *x* is an `int` or `float`, then a `float` with value equal to *e* raised to the power of *x* is returned. If *x* is a Scalar object, then a new *Scalar* is returned with an updated value and derivative(s).<jupyter_code>#exponential example
x = 1.0
print (ad.exp(x)) # should be 2.718281828459045<jupyter_output>2.718281828459045
{'fraction_non_alphanumeric': 0.13948566664528955, 'fraction_numerical': 0.02398512152889117, 'mean_word_length': 3.7557182067703567, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '615194', 'license': 'MIT', 'n_tokens_mistral': 4790, 'n_tokens_neox': 4515, 'n_words': 1969, 'path': 'docs/milestone2.ipynb', 'repo_name': 'cs207FinalProjectGroup/cs207-FinalProject', 'size': '15593'}