<commit_before>Design
=======

Here we delve into the core design underlying the Spark bolt array. Our considerations were based on the following observations having worked with ndarray-like objects in Spark:

- we usually want to parallelize operations across one or more axes, e.g. fitting a model to each of several data points, filtering each of several images, detrending each of several time series
- we want to easily and quickly change which axes we parallelize over
- we want to chain basic functional parallel operators with ndarray-style manipulations

In Spark, our primitive is a distributed collection of records, each of which takes the form of a ``key,value`` pair. We designed a ``BoltArraySpark`` object
in which the keys and values separately represent different dimensions of a single array. The keys are tuples that encode the indices of the "parallelized"
dimensions, while each value is a NumPy ``ndarray`` with one dimension for every "localized" dimension in the array. To help keep track of which dimensions
are "key-dimensions" and which are "value-dimensions", we impose a structure where the the key-dimensions always come before the value-dimensions. While this
choice does limit generality, we find that it is very useful for keeping track of which dimensions are parallelized and which are not -- an important distinction to
always keep in mind, as the efficiency of many operations depends on which type of dimensions they are applied to.. In line with this organization, we define an array's
"split" as the number of key-dimensions. This value is accessible through the ``split`` property.

Swapping
--------

One important operation for working with the ``BoltArraySpark`` is changing which axes are parallelized over. We call this operation "swapping" as it 
involves changing key-dimensions to value dimensions and vice-versa. Not only is this operation useful for users wishing to optimize distributed computations,
but it is also at the heart of transposing and reshaping these arrays. We expose this operation through the ``swap`` function. It takes a set of
key-dimensions that will be collected into value-dimensions, as well as a set of value-dimensions that will be broken up en route to
becoming key-dimensions. As we have decided that key-dimensions always precede valued-dimensions, this operation causes a corresponding transposition.
As a matter of convention, we specify that key-dimensions that are swapped to value-dimensions are placed immediately *after* the split, in the same order that they
appeared in the original array. On the other hand, the value-dimensions that move to the key-dimensions are placed immediately *before* the split (again,
in the same order in which they appeared in the original array).

This swapping operation can be very computational expensive -- changing how the data is distributed across the records can involve moving large volumes of data
across the Spark cluster. Thus, we have built on our experience working with these types of objects to make this operation as efficient as we can. The rationale
behind our swapping algorithm is perhaps best be understood by first thinking about two extreme approaches to solving this problem:

- At one extreme, one could collect every record to a single machine, construct the full array, slice it along the new set of key-dimensions,
and finally parallelize these new values to their own records. This approach obviously fails in a dramatic way when the array is too large to fit in memory on a single
machine. But even when working with a medium-sized dataset, it fails to leverage the distributed power of the Spark cluster.
- At the other end of the specturm, one could break the value (i.e subarray) in every record into singlton values, each tagged with its indices in the full array.
Then these singltons could be shuffled around the cluster, being sorted into new records based on the indices corresponding
to the new key-dimensions. Within these new records, the pieces could then be put back together into the correct array. Unlike the previous solution, this method
does take advantege of the distributed computing power of the Spark cluster. However, in breaking the data up into singleton values, it generates an extremely
large number of packets that must be shuffled around the network -- an strategy that is very costly.

Our algorithm seeks the happy medium between these two extremes. First, we break up the value subarrays into chunks, but only along the value-dimesions that are being
swapped to key-dimensions -- the other axes will remain together in the final result, so there is no need to break them apart only to have to put them<commit_msg>fixing bullets<commit_after>Design
=======

Here we delve into the core design underlying the Spark bolt array. Our considerations were based on the following observations having worked with ndarray-like objects in Spark:

- we usually want to parallelize operations across one or more axes, e.g. fitting a model to each of several data points, filtering each of several images, detrending each of several time series
- we want to easily and quickly change which axes we parallelize over
- we want to chain basic functional parallel operators with ndarray-style manipulations

In Spark, our primitive is a distributed collection of records, each of which takes the form of a ``key,value`` pair. We designed a ``BoltArraySpark`` object
in which the keys and values separately represent different dimensions of a single array. The keys are tuples that encode the indices of the "parallelized"
dimensions, while each value is a NumPy ``ndarray`` with one dimension for every "localized" dimension in the array. To help keep track of which dimensions
are "key-dimensions" and which are "value-dimensions", we impose a structure where the the key-dimensions always come before the value-dimensions. While this
choice does limit generality, we find that it is very useful for keeping track of which dimensions are parallelized and which are not -- an important distinction to
always keep in mind, as the efficiency of many operations depends on which type of dimensions they are applied to.. In line with this organization, we define an array's
"split" as the number of key-dimensions. This value is accessible through the ``split`` property.

Swapping
--------

One important operation for working with the ``BoltArraySpark`` is changing which axes are parallelized over. We call this operation "swapping" as it 
involves changing key-dimensions to value dimensions and vice-versa. Not only is this operation useful for users wishing to optimize distributed computations,
but it is also at the heart of transposing and reshaping these arrays. We expose this operation through the ``swap`` function. It takes a set of
key-dimensions that will be collected into value-dimensions, as well as a set of value-dimensions that will be broken up en route to
becoming key-dimensions. As we have decided that key-dimensions always precede valued-dimensions, this operation causes a corresponding transposition.
As a matter of convention, we specify that key-dimensions that are swapped to value-dimensions are placed immediately *after* the split, in the same order that they
appeared in the original array. On the other hand, the value-dimensions that move to the key-dimensions are placed immediately *before* the split (again,
in the same order in which they appeared in the original array).

This swapping operation can be very computational expensive -- changing how the data is distributed across the records can involve moving large volumes of data
across the Spark cluster. Thus, we have built on our experience working with these types of objects to make this operation as efficient as we can. The rationale
behind our swapping algorithm is perhaps best be understood by first thinking about two extreme approaches to solving this problem:

- At one extreme, one could collect every record to a single machine, construct the full array, slice it along the new set of key-dimensions, and finally parallelize these new values to their own records. This approach obviously fails in a dramatic way when the array is too large to fit in memory on a single machine. But even when working with a medium-sized dataset, it fails to leverage the distributed power of the Spark cluster.
- At the other end of the specturm, one could break the value (i.e subarray) in every record into singlton values, each tagged with its indices in the full array. Then these singltons could be shuffled around the cluster, being sorted into new records based on the indices corresponding to the new key-dimensions. Within these new records, the pieces could then be put back together into the correct array. Unlike the previous solution, this method does take advantege of the distributed computing power of the Spark cluster. However, in breaking the data up into singleton values, it generates an extremely large number of packets that must be shuffled around the network -- an strategy that is very costly.

Our algorithm seeks the happy medium between these two extremes. First, we break up the value subarrays into chunks, but only along the value-dimesions that are being
swapped to key-dimensions -- the other axes will remain together in the final result, so there is no need to break them apart only to have to put them
{'fraction_non_alphanumeric': 0.03555839454913233, 'fraction_numerical': 0.0, 'mean_word_length': 4.638655462184874, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5690072', 'n_tokens_mistral': 2096, 'n_tokens_neox': 2036, 'n_words': 1474}