<jupyter_start><jupyter_text>Logistic Regression Part 2--------------------------In this example, we extend the code from Part 1 with several important features:- Instead of just updating the weight matrix ``W``, we add a bias ``b`` and use the ``.variables()`` method to compactly update both variables.- We attach an additional computation to the transformer to compute the loss on a held-out validation dataset.- We switch from a flat ``C``-dimensional feature space to a ``W x H`` feature space to demonstrate multi-dimensional logistic regression.The corresponding jupyter notebook is found [here](https://github.com/NervanaSystems/ngraph-neon/blob/master/examples/walk_through/Logistic_Regression_Part_2.ipynb).<jupyter_code>import neon as ng
import neon.transformers as ngt
import gendata<jupyter_output><empty_output><jupyter_text>The axes creation is conceptually the same as before, except we now add a new axes ``H`` to represent the new feature space.<jupyter_code>ax_W = ng.make_axis(length=2)
ax_H = ng.make_axis(length=2)  # new axis added.
ax_N = ng.make_axis(length=128, name='N')<jupyter_output><empty_output><jupyter_text>Building the graphOur model, as in the previous example, has three placeholders: ``X``, ``Y``, and ``alpha``. But now, the the input ``X`` has shape ``(W, H, N)``:<jupyter_code>alpha = ng.placeholder(())
X = ng.placeholder([ax_W, ax_H, ax_N])  # now X has shape (W, H, N)
Y = ng.placeholder([ax_N])<jupyter_output><empty_output><jupyter_text>Similarly, the weight matrix is now multi-dimensional, with shape ``(W, H)``, and we add a new scalar bias variable. We want also to specify that, for the weight matrix ``W``, both axes will be reduced when computing the element-wise product and summation with the inputs (so we add ``-1`` to specify this)<jupyter_code>W = ng.variable([ax_W, ax_H], initial_value=0).named('W')  # now the Weight Matrix W has shape (W, H)
b = ng.variable((), initial_value=0).named('b')<jupyter_output><empty_output><jupyter_text>Our predicted output will now be including the bias ``b``. Please note there here the + operation implicitly broadcasts ``b`` to the batch size N, the size of the only axis of Y_hat:<jupyter_code>Y_hat = ng.sigmoid(ng.dot(W, X) + b)
L = ng.cross_entropy_binary(Y_hat, Y, out_axes=())<jupyter_output><empty_output><jupyter_text>For the parameter updates, instead of explicitly specifying the variables ``W`` and ``b``, we can call ``L.variables()`` to retrieve all the variables that the loss function depends on:<jupyter_code>print([var.name for var in L.variables()])<jupyter_output><empty_output><jupyter_text>For complicated ngraphs, the ``variables()`` method makes it easy to iterate over all its dependant variables. Our new parameter update is then:<jupyter_code>updates = [ng.assign(v, v - alpha * ng.deriv(L, v) / ng.batch_size(Y_hat))
           for v in L.variables()]<jupyter_output><empty_output><jupyter_text>Please note that this time we embedded the (call to the) gradient computation inside the definition of the weight update computation. As stated in the previous example, the ``ng.deriv`` function computes the backprop using autodiff. The update step computes the new weight and assigns it to ``W``:<jupyter_code>all_updates = ng.doall(updates)<jupyter_output><empty_output><jupyter_text>ComputationWe have our update computation as before, but we also add an evaluation computation that computes the loss on a separate dataset without performing the updates. Since the evaluation computation does not perform any update operation, we need not pass in the learning rate ``alpha`` For convenience, we define a function that computes the average cost across the validation set.<jupyter_code>def avg_loss(xs, ys):
    total_loss = 0
    for x, y in zip(xs, ys):
        loss_val = eval_fun(x, y)
        total_loss += loss_val
    return total_loss / x.shape[-1]<jupyter_output><empty_output><jupyter_text>We then generate our training and evaluation sets and perform the updates with the same technique that we used in the previous example. We emit the average loss on the validation set during training. Please note that because the length of the axes W and H is 2 now (for both; before we had only one axis of lenght 4), the number of weights is the same as in the previous example<jupyter_code>from contextlib import closing

with closing(ngt.make_transformer()) as transformer:

    update_fun = transformer.computation([L, W, b, all_updates], alpha, X, Y)
    eval_fun = transformer.computation(L, X, Y)
    
    g = gendata.MixtureGenerator([.5, .5], (ax_W.length, ax_H.length))
    XS, YS = g.gen_data(ax_N.length, 10)
    EVAL_XS, EVAL_YS = g.gen_data(ax_N.length, 4)

    print("Starting avg loss: {}".format(avg_loss(EVAL_XS, EVAL_YS)))
    for i in range(10):
        for xs, ys in zip(XS, YS):
            loss_val, w_val, b_val, _ = update_fun(5.0 / (1 + i), xs, ys)
        print("After epoch %d: W: %s, b: %s, avg loss %s" % (i, w_val.T, b_val, avg_loss(EVAL_XS, EVAL_YS)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10987482614742698, 'fraction_numerical': 0.004967216371945162, 'mean_word_length': 4.978622327790974, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '318034', 'license': 'Apache-2.0', 'n_tokens_mistral': 1609, 'n_tokens_neox': 1551, 'n_words': 594, 'path': 'examples/walk_through/Logistic_Regression_Part_2.ipynb', 'repo_name': 'MUTTERSCHIFF/ngraph-neon', 'size': '5033'}