<jupyter_start><jupyter_text>Differtless Documentation CS 107 Group 33: <NAME>, <NAME>, <NAME>, <NAME>  Introduction Differentiation of functions helps find the maxima, minima, and zero points that are essential for optimization and prediction in a variety of problems across a wide range of fields (i.e. Machine Learning, Physics, Fluid Mechanics, and more). Automatic differentiation (AD) is a computational approach that automatically finds the derivatives of a function using a computer program. For a given function, the AD software should be able to compute the derivatives of any order with machine precision. Through the computation of the Jacobian matrix and gradients, we are also able to optimize. Automatic differentiation differs from classical methods such as symbolic differentiation and numerical differentiation in that the AD method requires much less computational time and is not affected by numerical instability.Differtless is a software package for performing AD, with additional functionality for performing optimization to find stationary points of functions. Our goal is to make automatic differentiation effortless! BackgroundAutomatic differentiation is based on the decomposition of differentials using chain rule. Separated by the order of chain rule calculation, AD can be divided into two modes: forward mode and reverse mode. The forward mode calculates derivatives from the inner function to the outer function using chain rule repeatedly, while the reverse mode calculates from outside to inside. We can also say the forward mode calculates the product of the function's Jacobian matrix $J$ with the seed vectors specified by input variables, while reverse modes computes the product of the transpose of Jacobian matrix $J^T$ with  seed vectors. In a function with $m$ inputs and $n$ functions, represented by $R^m \rightarrow R^n$, forward mode is more efficient when $n$ >> $m$, while reverse mode is more efficient when $n$ << $m$.For our extension, we implemented optimization functionality for `numpy` and `scipy` mathematical operations, probability distributions, and spatial functions. `numpy` and `scipy` are popular open-source libraries for Python featuring a large collection of high-level numerical computing tools and modules for optimization, linear algebra, integration, interpolation, and special functions for tasks common in science and engineering.There are many useful applications of AD in this realm: for example, AD is often used within the context of Bayesian parameter inference for sampling from the posterior using methods like Hamiltonian Monte Carlo, where one needs to be able to take the gradients of probability distributions. Thus, we implemented the probability and cumulative distribution functions of commonly used statistical distributions such as the Normal, Poisson and Gamma distributions. When performing parameter inference in this manner, users often work with the logarithms of the functions to prevent overflow or underflow errors – thus we also implemented the log PDFs and log CDFs of all these distributions. We note that a lot of this functionality is not available in other AD packages like autograd.We hope that differtless will be a convenient way for users in a range of disciplines to integrate AD into their already-established workflows, without having to redefine functions they would typically use packages like numpy and scipy for, and without having to modify their pipeline to incorporate optimization routines.A computational graph is useful in visualizing the propagation of elementary operations that construct the function to be auto-differentiated. From here, it is easy to evaluate the numerical value after each operation, the derivatives of the elementary operations, and numerical values of derivatives with respect to each input variable.For an example function $$f(x,y)=\left[\begin{matrix}c_1 & c_2\end{matrix}\right]\left[\begin{matrix}w_{11} & w_{12} \\w_{21} & w_{22}\end{matrix}\right]\left[\begin{matrix}x  \\y\end{matrix}\right]$$<jupyter_code>from IPython.display import Image, display
display(Image(filename='img/comp.png', embed=True))
display(Image(filename='img/trace.png', embed=True))<jupyter_output><empty_output><jupyter_text>How to use DiffertlessDiffertless is available from PyPI via `pip install differtless``differtless` can also be obtained by `git clone https://github.com/differtless/cs107-FinalProject.git`If a user obtained `differtless` from github, they can install the package by running `pip install .`In both cases, package requirements are provided and checked for via `setup.py`      example github installation    >> git clone https://github.com/differtless/cs107-FinalProject.git    >> cd cs107-FinalProject    >> pip install .    Once the user has downloaded differtless they can run `from differtless import ad` and `import differtless.operations as op` in their driver script. Then when they have a function that they would like to automatically differentiate, they will: 1. Define the function, using either elementary operations (which we have overloaded- add, subtract, etc) or `differtless` operations for elementary functions (e.g. `op.sin`, `op.exp`)    * The user may also define multiple functions (NOTE: all functions must have the same parameters, regardless of whether they are used in the body of the function)2. User defines inputs and optional seed    * Scalar inputs:         * Single scalars may be in the form of a number or `list` containing one number        * Multiple scalars must be input as a `list` or `tuple`    * Vector inputs:        * Single and multiple vectors must both be defined as a list of lists or tuple of tuples        * Ex. `x = [[1,2,3]]`    * Seed must be in `list` form (or list of lists)3. User calls the forward function with their pre-defined function (or list of functions), inputs, and seeds.    * This includes the preprocessing step.    * NOTE: the seed matrix is optional, if no seed is specified the seed will be the appropriate identity matrix    * This will return the value and corresponding derivatives.4. User can also call `ad.minimize`, `ad.root`, or `ad.least_squares` to utilize optimization functionality     * users can also call on probability distributions (such as Normal, Poisson, Gaussian)  and the special functions overloaded in the `operations` module Example:     from differtless import ad    import differtless.operations as op     Define a function and provide inputs/seeds    def f1(x, y, z):        return x*y + op.exp(z)    def f2(x, y , z):        return (op.exp(x/y) + z)/y     scalar inputs    inputs = [1,2,3]    seeds = [[42, 1, 1], [2, 42, 2], [3, 3, 42]]     vector inputs    inputs = [[3, 2, 1], [4, 5, 6]]    seeds = [[1, 0], [0, 1]]     Run automatic differentiation    f1_ad = ad.forward(f1, inputs, seeds)     Run AD with vector-valued function    f1_f2_ad = ad.forward([f1, f2], inputs, seeds)     minimizing solution    min_sol = ad.minimize(f1, [2, 3, 4])    print(f'Function value: {f1_ad.value}')    print(f'Function derivatives: {f1_ad.gradients}')    print(f'Function Jacobian: {ad.Jacobian(f1, inputs)}')    print(f'Optimization minima solution: {min_sol}')  Full Functionality Demo: also available as demo.ipynb<jupyter_code>### import differtless ###
from differtless import ad
import differtless.operations as op

### define inputs of interest ###
# scalar input
single_scalar = [5]
# multiple scalar inputs (include seed)
multi_scalar = [2,3,4]
# vector input
single_vector = [[4,5,6]]
# multiple vector inputs
multi_vector = [[1,2,3],[4,5,6],[7,8,9]]
# seed matrix for multivariable functions
multi_seed = [[42,1,1],[2,42,1],[3,3,42]]

### define functions of interest ###
# single-variable functions
def single_f1(x):
    return op.exp(op.sin(x) ** 2)
def single_f2(x):
    return op.log(2 / x) - x
# multivariable functions
def multi_f1(x, y, z):
    return (z ** (x - op.tanh(x + y)))/x
def multi_f2(x, y, z):
    return x ** (10 - y)
### Preprocessing example ###
ad.preprocess(multi_vector)
### Scalar-Valued Functions ###
# Scalar input, single variable
forward1 = ad.forward(single_f1, single_scalar)
jacobian1 = ad.Jacobian(single_f1, single_scalar)
print(f'Value = {forward1.value}\nGradient = {forward1.gradients}\nJacobian = {jacobian1}')

# Vector input, single variable
forward2 = ad.forward(single_f1, single_vector)
jacobian2 = ad.Jacobian(single_f1, single_vector)
print(f'Value = {forward2.value}\nGradient = {forward2.gradients}\nJacobian = {jacobian2}')

# Scalar input, multivariable
forward3 = ad.forward(multi_f1, multi_scalar, multi_seed)
jacobian3 = ad.Jacobian(multi_f1, multi_scalar)
print(f'Value = {forward3.value}\nGradient = {forward3.gradients}\nJacobian = {jacobian3}')

# Vector input, multivariable
forward4 = ad.forward(multi_f1, multi_vector)
jacobian4 = ad.Jacobian(multi_f1, multi_vector)
print(f'Value = {forward4.value}\nGradient =\n{forward4.gradients}\nJacobian =\n{jacobian4}')
### Vector-Valued Functions ###
# Scalar input, single variable
forward5 = ad.forward([single_f1, single_f2], single_scalar)
jacobian5 = ad.Jacobian([single_f1, single_f2], single_scalar)
print(f'Value = {forward5.value}\nGradient = {forward5.gradients}\nJacobian = {jacobian5}')

# Scalar input, multivariable
forward6 = ad.forward([multi_f1, multi_f2], multi_scalar, multi_seed)
jacobian6 = ad.Jacobian([multi_f1, multi_f2], multi_scalar)
print(f'Value = {forward6.value}\nGradient =\n{forward6.gradients}\nJacobian =\n{jacobian6}')

# Vector input, single variable
forward7 = ad.forward(multi_f1, multi_vector, multi_seed)
jacobian7 = ad.Jacobian(multi_f1, multi_vector)
print(f'Value = {forward7.value}\nGradient =\n{forward7.gradients}\nJacobian =\n{jacobian7}')
### Extra Feature- Optimization ###

# Function optimization
ad.minimize(multi_f2, [2, 3, 4])

# Least-squares solution with bounds on the variables
ad.least_squares(single_f1, 4, bounds=(-4, 10))

# Root Finding
ad.root(single_f2, 1)
### Extra Feature - Probability Distributions ###

# Log CDF of Normal distribution
forward8 = ad.forward(op.Normal(loc=3, scale=2).logcdf, single_vector)
jacobian8 = ad.Jacobian(op.Normal(loc=3, scale=2).logcdf, single_vector)
print(f'Value = {forward8.value}\nGradient =\n{forward8.gradients}\nJacobian =\n{jacobian8}')

# Joint PDF of multiple variables described by different probability distributions
def joint_probabilities(x, y, z):
    return op.Normal(loc=1, scale=1).pdf(x) * op.Gamma(alpha=1, beta=2).pdf(y) * op.Poisson(mu=3).pmf(z)
forward9 = ad.forward(joint_probabilities, multi_vector, multi_seed)
print(f'Value = {forward9.value}\nGradient =\n{forward9.gradients}')

# Special Functions
def special(x):
    return 2*op.gamma(x) - op.erf(x/3)
forward10 = ad.forward(special, single_vector)
print(f'Value = {forward10.value}\nGradient =\n{forward10.gradients}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09321566464423607, 'fraction_numerical': 0.018018018018018018, 'mean_word_length': 5.195330296127563, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '401424', 'license': 'MIT', 'n_tokens_mistral': 3201, 'n_tokens_neox': 3018, 'n_words': 1239, 'path': 'docs/documentation.ipynb', 'repo_name': 'differtless/cs107-FinalProject', 'size': '10903'}