<jupyter_start><jupyter_text>TensorFlow Fold Quick StartTensorFlow Fold is a library for turning complicated Python data structures into TensorFlow Tensors.<jupyter_code># boilerplate
import random
import tensorflow as tf
sess = tf.InteractiveSession()
import tensorflow_fold as td<jupyter_output><empty_output><jupyter_text>The basic elements of Fold are *blocks*. We'll start with some blocks that work on simple data types.<jupyter_code>scalar_block = td.Scalar()
vector3_block = td.Vector(3)<jupyter_output><empty_output><jupyter_text>Blocks are functions with associated input and output types.<jupyter_code>def block_info(block):
    print("%s: %s -> %s" % (block, block.input_type, block.output_type))
    
block_info(scalar_block)
block_info(vector3_block)<jupyter_output><td.Scalar dtype='float32'>: PyObjectType() -> TensorType((), 'float32')
<td.Vector dtype='float32' size=3>: PyObjectType() -> TensorType((3,), 'float32')<jupyter_text>We can use `eval()` to see what a block does with its input:<jupyter_code>scalar_block.eval(42)
vector3_block.eval([1,2,3])<jupyter_output><empty_output><jupyter_text>Not very exciting. We can compose simple blocks together with `Record`, like so:<jupyter_code>record_block = td.Record({'foo': scalar_block, 'bar': vector3_block})
block_info(record_block)<jupyter_output><td.Record ordered=False>: PyObjectType() -> TupleType(TensorType((3,), 'float32'), TensorType((), 'float32'))<jupyter_text>We can see that Fold's type system is a bit richer than vanilla TF; we have tuple types! Running a record block does what you'd expect:<jupyter_code>record_block.eval({'foo': 1, 'bar': [5, 7, 9]})<jupyter_output><empty_output><jupyter_text>One useful thing you can do with blocks is wire them up to create pipelines using the `>>` operator, which performs function composition. For example, we can take our two tuple tensors and compose it with `Concat`, like so:<jupyter_code>record2vec_block = record_block >> td.Concat()
record2vec_block.eval({'foo': 1, 'bar': [5, 7, 9]})<jupyter_output><empty_output><jupyter_text>Note that because Python dicts are unordered, Fold always sorts the outputs of a record block by dictionary key. If you want to preserve order you can construct a Record block from an OrderedDict.The whole point of Fold is to get your data into TensorFlow; the `Function` block lets you convert a TITO (Tensors In, Tensors Out) function to a block:<jupyter_code>negative_block = record2vec_block >> td.Function(tf.negative)
negative_block.eval({'foo': 1, 'bar': [5, 7, 9]})<jupyter_output><empty_output><jupyter_text>This is all very cute, but where's the beef? Things start to get interesting when our inputs contain sequences of indeterminate length. The `Map` block comes in handy here:<jupyter_code>map_scalars_block = td.Map(td.Scalar())<jupyter_output><empty_output><jupyter_text>There's no TF type for sequences of indeterminate length, but Fold has one:<jupyter_code>block_info(map_scalars_block)<jupyter_output><td.Map element_block=<td.Scalar dtype='float32'>>: None -> SequenceType(TensorType((), 'float32'))<jupyter_text>Right, but you've done the TF [RNN Tutorial](https://www.tensorflow.org/tutorials/recurrent/) and even poked at [seq-to-seq](https://www.tensorflow.org/tutorials/seq2seq/). You're a wizard with [dynamic rnns](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn). What does Fold offer?Well, how about jagged arrays?<jupyter_code>jagged_block = td.Map(td.Map(td.Scalar()))
block_info(jagged_block)<jupyter_output><td.Map element_block=<td.Map element_block=<td.Scalar dtype='float32'>>>: None -> SequenceType(SequenceType(TensorType((), 'float32')))<jupyter_text>The Fold type system is fully compositional; any block you can create can be composed with `Map` to create a sequence, or `Record` to create a tuple, or both to create sequences of tuples or tuples of sequences:<jupyter_code>seq_of_tuples_block = td.Map(td.Record({'foo': td.Scalar(), 'bar': td.Scalar()}))
seq_of_tuples_block.eval([{'foo': 1, 'bar': 2}, {'foo': 3, 'bar': 4}])
tuple_of_seqs_block = td.Record({'foo': td.Map(td.Scalar()), 'bar': td.Map(td.Scalar())})
tuple_of_seqs_block.eval({'foo': range(3), 'bar': range(7)})<jupyter_output><empty_output><jupyter_text>Most of the time, you'll eventually want to get one or more tensors out of your sequence, for wiring up to your particular learning task. Fold has a bunch of built-in reduction functions for this that do more or less what you'd expect:<jupyter_code>((td.Map(td.Scalar()) >> td.Sum()).eval(range(10)),
 (td.Map(td.Scalar()) >> td.Min()).eval(range(10)),
 (td.Map(td.Scalar()) >> td.Max()).eval(range(10)))<jupyter_output><empty_output><jupyter_text>The general form of such functions is `Reduce`:<jupyter_code>(td.Map(td.Scalar()) >> td.Reduce(td.Function(tf.multiply))).eval(range(1,10))<jupyter_output><empty_output><jupyter_text>If the order of operations is important, you should use `Fold` instead of `Reduce` (but if you can use `Reduce` you should, because it will be faster):<jupyter_code>((td.Map(td.Scalar()) >> td.Fold(td.Function(tf.divide), tf.ones([]))).eval(range(1,5)),
 (td.Map(td.Scalar()) >> td.Reduce(td.Function(tf.divide), tf.ones([]))).eval(range(1,5)))  # bad, not associative!<jupyter_output><empty_output><jupyter_text>Now, let's do some learning! This is the part where "magic" happens; if you want a deeper understanding of what's happening here you might want to jump right to our more formal [blocks tutorial](https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/blocks.md) or learn more about [running blocks in TensorFlow](https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/running.md)<jupyter_code>def reduce_net_block():
    net_block = td.Concat() >> td.FC(20) >> td.FC(1, activation=None) >> td.Function(lambda xs: tf.squeeze(xs, axis=1))
    return td.Map(td.Scalar()) >> td.Reduce(net_block)<jupyter_output><empty_output><jupyter_text>The `reduce_net_block` function creates a block (`net_block`) that contains a two-layer fully connected (FC) network that takes a pair of scalar tensors as input and produces a scalar tensor as output. This network gets applied in a binary tree to reduce a sequence of scalar tensors to a single scalar tensor.One thing to notice here is that we are calling [`tf.squeeze`](https://www.tensorflow.org/versions/r1.0/api_docs/python/array_ops/shapes_and_shapingsqueeze) with `axis=1`, even though the Fold output type of `td.FC(1, activation=None)` (and hence the input type of the enclosing `Function` block) is a `TensorType` with shape `(1)`. This is because all Fold blocks actually run on TF tensors with an implicit leading batch dimension, which enables execution via [*dynamic batching*](https://arxiv.org/abs/1702.02181). It is important to bear this in mind when creating `Function` blocks that wrap functions that are not applied elementwise.<jupyter_code>def random_example(fn):
    length = random.randrange(1, 10)
    data = [random.uniform(0,1) for _ in range(length)]
    result = fn(data)
    return data, result<jupyter_output><empty_output><jupyter_text>The `random_example` function generates training data consisting of `(example, fn(example))` pairs, where `example` is a random list of numbers, e.g.:<jupyter_code>random_example(sum)
random_example(min)
def train(fn, batch_size=100):
    net_block = reduce_net_block()
    compiler = td.Compiler.create((net_block, td.Scalar()))
    y, y_ = compiler.output_tensors
    loss = tf.nn.l2_loss(y - y_)
    train = tf.train.AdamOptimizer().minimize(loss)
    sess.run(tf.global_variables_initializer())
    validation_fd = compiler.build_feed_dict(random_example(fn) for _ in range(1000))
    for i in range(2000):
        sess.run(train, compiler.build_feed_dict(random_example(fn) for _ in range(batch_size)))
        if i % 100 == 0:
            print(i, sess.run(loss, validation_fd))
    return net_block<jupyter_output><empty_output><jupyter_text>Now we're going to train a neural network to approximate a reduction function of our choosing. Calling `eval()` repeatedly is super-slow and cannot exploit batch-wise parallelism, so we create a [`Compiler`](https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/py/td.mdcompiler). See our page on [running blocks in TensorFlow](https://github.com/tensorflow/fold/blob/master/tensorflow_fold/g3doc/running.md) for more on Compilers and how to use them effectively.<jupyter_code>sum_block = train(sum)
sum_block.eval([1, 1])<jupyter_output><empty_output><jupyter_text>Breaking news: deep neural network learns to calculate 1 + 1!!!! Of course we've done something a little sneaky here by constructing a model that can only represent associative functions and then training it to compute an associative function. The technical term for being sneaky in machine learning is [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias).<jupyter_code>min_block = train(min)
min_block.eval([2, -1, 4])<jupyter_output><empty_output><jupyter_text>Oh noes! What went wrong? Note that we trained our network to compute `min` on positive numbers; negative numbers are outside of its input distribution.<jupyter_code>min_block.eval([0.3, 0.2, 0.9])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1245266688304663, 'fraction_numerical': 0.013415557719355188, 'mean_word_length': 6.365737051792829, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 121, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '10358', 'license': 'Apache-2.0', 'n_tokens_mistral': 3055, 'n_tokens_neox': 2922, 'n_words': 872, 'path': 'tensorflow_fold/g3doc/quick.ipynb', 'repo_name': 'mudrat/fold', 'size': '9243'}