<commit_before>.. _tutorial:
.. _zarr package: https://zarr.readthedocs.io/en/stable/index.html
.. _Nd4j: https://deeplearning4j.org/docs/latest/nd4j-overview

Tutorial
========
JZarr provides classes and functions to handle N-dimensional arrays data
whose data can be divided into chunks and each chunk can be compressed.
If you are already familiar with the python `zarr package`_ then JZarr
provide similar functionality, but without NumPy array behavior.

If you need array objects which behave almost like NumPy arrays you can wrap the data
using ND4J INDArray `from deeplearning4j.org <Nd4j>`_.
You can find examples in the data writing and reading examples below.

Alternatively you can use :code:`ucar.ma2.Array` from `netcdf-java Common Data Model
<https://github.com/Unidata/netcdf-java/blob/master/README.md>`_ to wrap the data.

.. _tutorial_create:

Creating an array
-----------------
JZarr has several functions for creating arrays. For example:

.. highlight:: java

.. literalinclude:: ./examples/java/Tutorial.java
  :caption: `example 1 from Tutorial.java <https://github.com/bcdev/jzarr/blob/master/docs/examples/java/Tutorial.java>`_
  :start-after: void example_1
  :end-before: createOutput
  :dedent: 8

.. highlight:: none

A :code:`System.out.println(array);` then creates the following output

.. literalinclude:: ./examples/output/Tutorial_example_1.txt

The code above creates a 2-dimensional array of 32-bit integers with 10000 rows and 10000 columns,
divided into chunks where each chunk has 1000 rows and 1000 columns (and so there will be 100 chunks in total).

For a complete list of array creation routines see the :ref:`array creation <array_creation>` module documentation.

.. _tutoral_writing_and_reading_data:

Writing and reading data
------------------------
This example shows how to write and read a region to an array.

Creates an array with size (rows=5, cols=7), primitive datatype :code:`int` and with a fill value of :code:`-9999`.

.. highlight:: java

.. literalinclude:: ./examples/java/Tutorial.java
  :caption: `example 2 from Tutorial.java <https://github.com/bcdev/jzarr/blob/master/docs/examples/java/Tutorial.java>`_
  :start-after: example 2 code snippet 1 begin
  :end-before: example 2 code snippet 1 end
  :dedent: 8

Prepare data which should be written with a shape of (3, 5) and an offset of (1, 1) to the center of the array.

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 2 begin
  :end-before: example 2 code snippet 2 end
  :dedent: 8

Write the data

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 3 begin
  :end-before: example 2 code snippet 3 end
  :dedent: 8

Read the entire data from the array (int[] with size 5 * 7)

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 4 begin
  :end-before: example 2 code snippet 4 end
  :dedent: 8

| Print out the data red.
| We can see the data written before is surrounded by the fill value :code:`-9999`.

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 5 begin
  :end-before: example 2 code snippet 5 end
  :dedent: 8

.. highlight:: none

Creates the following output

.. literalinclude:: ./examples/output/Tutorial_example_2.txt

| The output displays that the data is written (with an offset of [1, 1]) to the center of the array.
| The written data is surrounded by a :code:`-9999` value border which is the fill value defined above.

.. note::

   `Nd4j <Nd4j>`_ is not part of the JZarr library. It is only used in this showcase to demonstrate how the data can be used.

.. _tutoral_persistent_arrays:

Persistent arrays
-----------------
In the examples above, compressed data (default compressor) for each chunk of the array was stored
in main memory. JZarr arrays can also be stored on a file system, enabling persistence of data
between sessions. For example:

.. highlight:: java<commit_msg>Prepare documentation<commit_after>.. _tutorial:
.. _zarr package: https://zarr.readthedocs.io/en/stable/index.html
.. _Nd4j: https://deeplearning4j.org/docs/latest/nd4j-overview

Tutorial
========
JZarr provides classes and functions to handle N-dimensional arrays data
whose data can be divided into chunks and each chunk can be compressed.
If you are already familiar with the python `zarr package`_ then JZarr
provide similar functionality, but without NumPy array behavior.

If you need array objects which behave almost like NumPy arrays you can wrap the data
using ND4J INDArray `from deeplearning4j.org <Nd4j>`_.
You can find examples in the data writing and reading examples below.

Alternatively you can use :code:`ucar.ma2.Array` from `netcdf-java Common Data Model
<https://github.com/Unidata/netcdf-java/blob/master/README.md>`_ to wrap the data.

.. _tutorial_create:

Creating an array
-----------------
JZarr has several functions for creating arrays. For example:

.. highlight:: java

.. literalinclude:: ./examples/java/Tutorial.java
  :caption: `example 1 from Tutorial.java <https://github.com/bcdev/jzarr/blob/master/docs/examples/java/Tutorial.java>`_
  :start-after: void example_1
  :end-before: createOutput
  :dedent: 8

.. highlight:: none

A :code:`System.out.println(array);` then creates the following output

.. literalinclude:: ./examples/output/Tutorial_example_1.txt

The code above creates a 2-dimensional array of 32-bit integers with 10000 rows and 10000 columns,
divided into chunks where each chunk has 1000 rows and 1000 columns (and so there will be 100 chunks in total).

For a complete list of array creation routines see the :ref:`array creation <array_creation>` module documentation.

.. _tutoral_writing_and_reading_data:

Writing and reading data
------------------------
This example shows how to write and read a region to an array.

Creates an array with size (rows=5, cols=7), primitive datatype :code:`int` and with a fill value of :code:`-9999`.

.. highlight:: java

.. literalinclude:: ./examples/java/Tutorial.java
  :caption: `example 2 from Tutorial.java <https://github.com/bcdev/jzarr/blob/master/docs/examples/java/Tutorial.java>`_
  :start-after: example 2 code snippet 1 begin
  :end-before: example 2 code snippet 1 end
  :dedent: 8

Prepare data which should be written with a shape of (3, 5) and an offset of (1, 1) to the center of the array.

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 2 begin
  :end-before: example 2 code snippet 2 end
  :dedent: 8

Write the data

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 3 begin
  :end-before: example 2 code snippet 3 end
  :dedent: 8

Read the entire data from the array (int[] with size 5 * 7)

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 4 begin
  :end-before: example 2 code snippet 4 end
  :dedent: 8

Print out the data red.

.. literalinclude:: ./examples/java/Tutorial.java
  :start-after: example 2 code snippet 5 begin
  :end-before: example 2 code snippet 5 end
  :dedent: 8

.. highlight:: none

Creates the following output

.. literalinclude:: ./examples/output/Tutorial_example_2.txt

The output displays that the data written before (written with an offset of [1, 1]) is surrounded by the fill value :code:`-9999`.

.. note::

   `Nd4j <Nd4j>`_ is not part of the JZarr library. It is only used in this showcase to demonstrate how the data can be used.

.. _tutoral_persistent_arrays:

Persistent arrays
-----------------
In the examples above, compressed data (default compressor) for each chunk of the array was stored
in main memory. JZarr arrays can also be stored on a file system, enabling persistence of data
between sessions. For example:

.. highlight:: java
{'fraction_non_alphanumeric': 0.10428662827895073, 'fraction_numerical': 0.021497120921305183, 'mean_word_length': 3.795092024539877, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1021252', 'n_tokens_mistral': 2490, 'n_tokens_neox': 2240, 'n_words': 963}