<jupyter_start><jupyter_text>Not only is the result accurate (the precision is only limited by the floatingpoint errors), but the ```gradient()``` method only goes through the recorded computations once (in reverse order), no matter how many variables there are, so it is incredibly efficient. It’s like magic!Only put the strict minimum inside the ```tf.GradientTape()``` block,to save memory. Alternatively, you can pause recording by creatinga ```with tape.stop_recording()``` block inside the ```tf.GradientTape()``` block.**The tape is automatically erased immediately after you call its gradient() method, soyou will get an exception if you try to call gradient() twice.**<jupyter_code>with tf.GradientTape() as tape:
    z = f(w1, w2)
dz_dw1 = tape.gradient(z, w1) # => tensor 36.0
print("dz_dw1 :", dz_dw1)
dz_dw2 = tape.gradient(z, w2) # RuntimeError!
print("dz_dw2: ", dz_dw1)<jupyter_output>dz_dw1 : tf.Tensor(36.0, shape=(), dtype=float32)<jupyter_text>**If you need to call ```gradient()``` more than once, you must make the tape persistent,and delete it when you are done with it to free resources:**<jupyter_code>with tf.GradientTape(persistent=True) as tape:
    z = f(w1, w2)
dz_dw1 = tape.gradient(z, w1) # => tensor 36.0
print("dz_dw1 :", dz_dw1)
dz_dw2 = tape.gradient(z, w2) # RuntimeError!
print("dz_dw2: ", dz_dw1)
del tape<jupyter_output>dz_dw1 : tf.Tensor(36.0, shape=(), dtype=float32)
dz_dw2:  tf.Tensor(36.0, shape=(), dtype=float32)<jupyter_text>**By default, the tape will only track operations involving variables, so if you try tocompute the gradient of z with regards to anything else than a variable, the result willbe None.** However, you can force the tape to watch any tensors you like, to record every operation that involves them. You can then compute gradients with regards to these tensors, as if they were variables.<jupyter_code>c1, c2 = tf.constant(5.), tf.constant(3.)
with tf.GradientTape() as tape:
    z = f(c1, c2)
gradients = tape.gradient(z, [c1, c2]) # returns [None, None]
print(gradients)
with tf.GradientTape() as tape:
    tape.watch(c1)
    tape.watch(c2)
    z = f(c1, c2)
gradients = tape.gradient(z, [c1, c2])
print("dc1 : {} \t dc2 : {} ".format(gradients[0].numpy(), gradients[1].numpy()))<jupyter_output>dc1 : 36.0 	 dc2 : 10.0<jupyter_text>This can be useful in some cases, for example if you want to implement a regulariza‐tion loss that penalizes activations that vary a lot when the inputs vary little: the losswill be based on the gradient of the activations with regards to the inputs. Since theinputs are not variables, you would need to tell the tape to watch them. **In some rare cases you may want to stop gradients from backpropagating throughsome part of your neural network. To do this, you must use the ```tf.stop_gradient()``` function: it just returns its inputs during the forward pass (like ```tf.identity()```), but it does not let gradients through during backpropagation (it acts like a constant).**For example:<jupyter_code>def f(w1, w2):
    return 3 * w1 ** 2 + tf.stop_gradient(2 * w1 * w2)

with tf.GradientTape() as tape:
    z = f(w1, w2) # same result as without stop_gradient()
    
gradients = tape.gradient(z, [w1, w2]) # => returns [tensor 30., None]

print(gradients)<jupyter_output>[<tf.Tensor: id=1083, shape=(), dtype=float32, numpy=30.0>, None]<jupyter_text>you may occasionally run into some numerical issues when computing gradients. For example, if you compute the gradients of the my_softplus() function forlarge inputs, the result will be NaN: This is because computing the gradients of this function using autodiff leads to somenumerical difficulties: due to floating point precision errors, autodiff ends up computing infinity divided by infinity (which returns NaN). Fortunately, we can analytically find that the derivative of the softplus function is just $ 1 / (1 + 1 / exp(x))$, which is numerically stable. Next, we can tell TensorFlow to use this stable function whencomputing the gradients of the my_softplus() function, by decorating it with``` @tf.custom_gradient ``` , and making it return both its normal output and the functionthat computes the derivatives (note that it will receive as input the gradients that werebackpropagated so far, down to the softplus function, and according to the chain rulewe should multiply them with this function’s gradients)<jupyter_code>@tf.custom_gradient
def my_better_softplus(z):
    exp = tf.exp(z)
    def my_softplus_gradients(grad):
        return grad / (1 + 1 / exp)
    return tf.math.log(exp + 1), my_softplus_gradients<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10205850487540628, 'fraction_numerical': 0.02145178764897075, 'mean_word_length': 5.105820105820106, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '69490', 'license': 'MIT', 'n_tokens_mistral': 1454, 'n_tokens_neox': 1373, 'n_words': 579, 'path': '27_computing_gradients_using_autodiff.ipynb', 'repo_name': 'renjithbaby23/tf2.0_examples', 'size': '4615'}