<commit_before>Datashape
=========

.. highlight:: erlang

Datashape is a generalization of ``dtype`` and ``shape`` into a micro
type system which lets us overlay high level structure on existing
data in Table and Array objects that can inform better code
generation and scheduling.

Overview
~~~~~~~~

Just like in traditional NumPy the preferred method of implementing
generic vector operators is through ad-hoc polymorphism. Numpy's style
of ad-hoc polymorphism allows ufunc objects to have different behaviors
when "viewed" at different types. The runtime system then chooses an
appropriate implementation for each application of the function, based
on the types of the arguments. Blaze simply extends this specialization
to data structure and data layout as well as data type ( dtype ).

In fact many of the ideas behind datashape are generalizations and
combinations of notions found in Numpy:

+----------------+----------------+
| Numpy          | Blaze          |
+================+================+
| Broadcasting   | Unification    |
+----------------+----------------+
| Shape          |                |
+----------------+ Datashape      |
| Dtype          |                |
+----------------+----------------+
| Ufunc          | Gufunc         |
+----------------+----------------+

Datashapes in Blaze do not form a hierarchy or permit subtyping. This
differs from type systems found in other languages like OCaml and Julia
which achieve a measure of type polymorphism through the construction of
hierarchies of types with an explicit pre-ordering.

Blaze favors the other approach in that types do not exist in a
hierarchy but instead are inferred through constraint generation at
"compile time". In addition it also permits a weakened version of
gradual typing through a dynamic type ( denoted ``?`` ) which will allow
a "escape hatch" in the type system for expressing types of values which
cannot be known until runtime.

The goal of next generation of vector operations over Blaze structures
aim to allow a richer and more declarative way of phrasing operations
over semi-structured data. While types are a necessary part of writing
efficient code, the ideal type system is one which disappears entirely!

Machine Types
~~~~~~~~~~~~~

+----------------+
| long           |
+----------------+
| double         |
+----------------+
| short          |
+----------------+
| longdouble     |
+----------------+
| char           |
+----------------+
| int8           |
+----------------+
| int16          |
+----------------+
| int32          |
+----------------+
| int64          |
+----------------+
| uint           |
+----------------+
| ulong          |
+----------------+
| ulonglong      |
+----------------+
| uint8          |
+----------------+
| uint16         |
+----------------+
| uint32         |
+----------------+
| uint64         |
+----------------+
| float8         |
+----------------+
| float16        |
+----------------+
| float32        |
+----------------+
| float64        |
+----------------+
| float128       |
+----------------+
| complex64      |
+----------------+
| complex128     |
+----------------+
| complex256     |
+----------------+
| void           |
+----------------+
| bool           |
+----------------+
| pyobj          |<commit_msg>Added shorthand, removed nonsensical datashapes<commit_after>Datashape
=========

.. highlight:: erlang

Datashape is a generalization of ``dtype`` and ``shape`` into a micro
type system which lets us overlay high level structure on existing
data in Table and Array objects that can inform better code
generation and scheduling.

Overview
~~~~~~~~

Just like in traditional NumPy the preferred method of implementing
generic vector operators is through ad-hoc polymorphism. Numpy's style
of ad-hoc polymorphism allows ufunc objects to have different behaviors
when "viewed" at different types. The runtime system then chooses an
appropriate implementation for each application of the function, based
on the types of the arguments. Blaze simply extends this specialization
to data structure and data layout as well as data type ( dtype ).

In fact many of the ideas behind datashape are generalizations and
combinations of notions found in Numpy:

+----------------+----------------+
| Numpy          | Blaze          |
+================+================+
| Broadcasting   | Unification    |
+----------------+----------------+
| Shape          |                |
+----------------+ Datashape      |
| Dtype          |                |
+----------------+----------------+
| Ufunc          | Gufunc         |
+----------------+----------------+

Datashapes in Blaze do not form a hierarchy or permit subtyping. This
differs from type systems found in other languages like OCaml and Julia
which achieve a measure of type polymorphism through the construction of
hierarchies of types with an explicit pre-ordering.

Blaze favors the other approach in that types do not exist in a
hierarchy but instead are inferred through constraint generation at
"compile time". In addition it also permits a weakened version of
gradual typing through a dynamic type ( denoted ``?`` ) which will allow
a "escape hatch" in the type system for expressing types of values which
cannot be known until runtime.

The goal of next generation of vector operations over Blaze structures
aim to allow a richer and more declarative way of phrasing operations
over semi-structured data. While types are a necessary part of writing
efficient code, the ideal type system is one which disappears entirely!

Machine Types
~~~~~~~~~~~~~

+----------------+
| long           |
+----------------+
| double         |
+----------------+
| short          |
+----------------+
| longdouble     |
+----------------+
| char           |
+----------------+
| int8           |
+----------------+
| int16          |
+----------------+
| int32          |
+----------------+
| int64          |
+----------------+
| uint           |
+----------------+
| uint8          |
+----------------+
| uint16         |
+----------------+
| uint32         |
+----------------+
| uint64         |
+----------------+
| float16        |
+----------------+
| float32        |
+----------------+
| float64        |
+----------------+
| float128       |
+----------------+
| complex64      |
+----------------+
| complex128     |
+----------------+
| complex256     |
+----------------+
| void           |
+----------------+
| bool           |
+----------------+
| pyobj          |
{'fraction_non_alphanumeric': 0.24546572394712574, 'fraction_numerical': 0.009683369197663695, 'mean_word_length': 1.5113855654187571, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2851295', 'n_tokens_mistral': 1536, 'n_tokens_neox': 1415, 'n_words': 633}