<jupyter_start><jupyter_text>GPU-Accelerated Numerical Computing with MatX  Tutorial List1. Introduction (this tutorial)2. [Operators](02_operators.ipynb)3. [Executors](03_executors.ipynb)4. [Radar Pipeline Example](04_radar_pipeline.ipynb)  IntroductionWelcome to NVIDIA's MatX training! In this course, you will learn how to use MatX, a modern C++ header-only library designed for generic matrix and n-dimensional array (tensor) operations. By borrowing heavily from both Python and MATLAB syntax, MatX is designed to provide native CUDA performance all behind a friendly and extensible API. Whereas other GPU-accelerated numerical computing libraries like [CuPy](https://cupy.dev/) and [Numba](http://numba.pydata.org/) allow the Python developer to build GPU-centric applications, MatX extends this "quick build, fast performance" thesis to the C++ developer. Our ultimate goal is to encourage developer productivity and provide a quick on-ramp from prototype code to statically-typed production code.  Training StructureThis training contains a series of tutorials in increasing order of name that will guide you through basic and intermediate MatX features. Most tutorials will require you to open the source code, make a small change, and run it from the Jupyter notebook. As time permits, there is another notebook called `99_assignments.ipynb` that will give a series of problems to solve using MatX primitives. The assignments have verification code to make sure the answer is correct.  MotivationWriting high-performance GPU code usually requires writing CUDA code directly and having at least a basic knowledge of the underlying hardware. Users not wanting to invest the time to learn this will typically use higher-level languages with CUDA bindings, such as MATLAB or Python. Using these high-level languages may cause interoperability with existing libraries and software more difficult. High-level languages can also cause a performance penalty for applications that require the lowest latency or highest throughput. MatX aims to provide syntax familiar to high-level languages with the performance and interoperability of C++. Most of the runtime costs are transferred into compile-time costs by using C++ templates, and kernels are auto-generated for most tensor and domain-specific functions. MatX does not require any CUDA or hardware knowledge; users can write algebraic expressions or use simple functions on tensor objects without every writing CUDA code.  MatX StructureMatX is structured into two separate, but compatible, APIs: tensor and frontend. The tensor API provides a suite of mathematical operations for operating on MatX tensor types. These operations are evaluated at compile-time and can be combined into chained expressions to improve performance. The frontend APIs are simple interfaces into numerous existing CUDA libraries. These include cuFFT, cuBLAS, cuSolver, and more. A common tensor type can be passed seamlessly between the tensor and frontend APIs, and MatX uses the information from these operations to call the appropriate kernels or libraries.One of the key concepts in MatX's design and usage is the disassociation of memory allocation from processing. We will cover this concept in depth during the training, but MatX optimizes primitive performance by discouraging operations that negatively impact performance, such as deep copies. When creating an application, the user creates views into memory regions that provide zero-cost abstractions of looking at the data differently in memory.  TerminologyThe basic building block of all MatX operations is the tensor. Having the common tensor type for all operations provides a powerful tool for manipulating data, and more importantly, abstracts much of the complexity from the user. MatX does not use traditional tensor notation (Einstein), and instead treats a tensor simply as scalars (0-dimensional tensors), vectors (1-dimensional tensors), or multi-dimensonal matrices (n-dimensional tensors). Tensors have three main properties: rank, size, and type. The rank and type are specified at compile time, while the size can be specified at either runtime or compile-time. The rank describes how many dimensions the tensor has. The size of the tensor describes how many elements are in each dimension. Lastly, the type of the tensor is the type of data stored in the tensor. Different data types will change the size and capabilities of the tensor.- **Tensors**    Tensors (`tensor_t`) provide views of the underlying tensor data and optionally allocate managed memory. Tensors describe properties such as stride, size of each dimension, and provide basic accessor functions for retrieving the data in a view. Tensors can be created from other tensors when convenient, which allows a view into another tensor without copying any data. For example, a sliced view of a tensor (which we'll get to soon) can be used to generate two more tensors of both real and imaginary parts of a complex number. The term `view` is sometimes interchangeably used with the term `tensor` in MatX since all tensors are simply a view into underlying memory. The memory backing tensors is reference counted; the last tensor to be destroyed either explicitly or implicitly will free any non-user-managed data.- **Operators**    Operators are an abstract term used for types that can return a value at a given index. Currently there are three types of operators: tensors, operator expressions, and generators. Operators are covered in more detail in lesson 2.  Our First ExampleIn this example, we give an introduction to the MatX library by performing basic operations on tensor objects. We will show the following operations on tensors:0. Library Import1. Creation2. Initialization3. Permuting (Rearrange dimensions of tensor)4. Slicing5. CloningAll of these operations are on tensor object types only, and do not use any of the frontend API. If desired, open `example1.cu` in a separate tab to view the entire file. Let's walk through the example line-by-line.  0. Library ImportDuring this tutorial, we will be writing our MatX code in a CUDA file (`.cu`), compiled via the CUDA Compiler, `nvcc`. If you're curious, the specific command line to build and execute code can be found [here](exercises/compile_and_run.sh).When using MatX, be sure to import the library via:```c++include using namespace matx;```  1. CreationFirst, we create a `tensor_t` object:```c++tensor_t t2({5,4});```tensor_t objects take two template parameters to instantiate: 1. The type of the underlying data2. The rank of the tensorOn this line, we are creating a 2D tensor (matrix) of integer values. These template parameters are required for all tensor objects to allow compile-time optimizations. The constructor of `tensor_t` takes a `tensorShape_t` object to specify the size of each of the two dimensions. Creating a shape can be done with an initializer list as well, so we construct the shape object using this shortened format. In this case, we are asking for a 5x4 2D tensor, where 5 is the number of rows and 4 is the number of columns.**NOTE** Unlike MATLAB, MatX follows the C-style for indexing, meaning we assume row-major formats rather than column-major, and 0-based indexing rather than 1-based. By not passing extra arguments, we are asking the tensor object to allocate the memory using CUDA managed memory so that it is visible by both host and device code. Alternatively, users can opt to manage their own memory using other forms of the constructor (see documentation).  2. Initialization After allocating the tensor, we initialize the underlying data:```c++t2.SetVals({  {1, 2, 3, 4},        {5, 6, 7, 8},        {9, 10, 11, 12},        {13, 14, 15, 16},        {17, 18, 19, 20}});t2.PrefetchDevice(0);```The tensor is initialized using a nested initializer list inside of the `SetVals` member function, specifying the values of the matrix. The initializer list is a single-nested list to match a 2D tensor shape, but this can be extended up to 4D tensors. `operator()` is also available to set and get individual values of a tensor as an alternative:```c++   t2(0,0) = 1;   t2(0,1) = 2;   ...```The next call to `PrefetchDevice(0)` ensures that any data that may have been set on the host is now copied to the device. The `0` parameter is the CUDA stream, and for this example, we're operating only with the default stream (0). By default, prefetching will instruct the CUDA runtime to prefetch all data visible to the current view. After this step completes, the initialized data is visible on both the host and device. Note that if the user is not using managed memory, `operator()` to set values is not available on the host, and instead the values should be copied from a host-side tensor. The prefetch line can be ommited and the program will still work correctly, but the first time `t2` is accesed on the device will trigger a page fault, and the data would be moved automatically later.On the next line we print the size of the tensor dimensions and the current data inside the tensor:```t2.Print();```  `Print` is a utility function of a tensor to print a tensor's contents to stdout. Printing tensors can be used with data that resides either on the host or device, and MatX will determine this at runtime. With no arguments it will print the entire contents of the tensor. However, the size of the printing can also be limited by passing a limit to each dimension. For example, `Print(3,2)` would print the first 2 columns and 3 rows of the 2D tensor. The contents of the tensor printed should appear as an increasing sequence of numbers from the top to bottom rows.Open the file [exercises/example1_init.cu](exercises/example1_init.cu) and edit the contents where you see TODO markers.<jupyter_code>!./exercises/compile_and_run.sh example1_init<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000:    1    2    3    4 000001:    5    6    7    8 000002:    9   10   11   12 000003:   13   14   15   16 000004:   17   18   19   20 ```  3. PermuteThe next section calls `Permute` on the returned view:```c++t2p = t2.Permute({1,0});t2p.Print();````Permute` returns a view of the data with the dimensions swapped to match the order of the initializer list argument. In this case there are only two dimensions being permuted on a 2D tensor, so it's equivalent to a matrix transpose. However, `Permute` can be used on higher-order tensors with the dimensions swapped in any particular order. Observe the data and size of the tensor is now transposed when using this view:![Permuted/Transposed 2D Tensor](img/dli-transpose.png)Open the file [exercises/example1_permute.cu](exercises/example1_permute.cu) and edit the contents where you see TODO markers.<jupyter_code>!./exercises/compile_and_run.sh example1_permute<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000: 1 5 9 13 17 000001: 2 6 10 14 18 000002: 3 7 11 15 19 000003: 4 8 12 16 20 ``` Note that none of the underlying data has been modified. A permuted view simply accesses the data as if it were transposed, but the actual order of data in memory has not changed. This can be confirmed by re-printing the previous non-transposed view. By not modifying the data in memory, a permuted view *may* be slower to access than a non-permuted view with contiguous entries. In general, permuted views are useful for infrequent accesses, but if the transposed data would be accessed repeatedly it may be faster to permute the data in memory using the `transpose` operator.  4. SliceThe next line takes a slice of the 2D tensor by selecting a subset of data in both dimensions:```c++t2s = t2.Slice({1,1}, {3, 3});````t2s` is now a view of the same data, but starting at index 1 and ending at index 3 (exclusive) on both dimensions. This is equivalent to Python using `t2[1:3, 1:3]`. Since a new sliced view is returned, the new view will have dimensions `{2, 2}`.![2D Slice](img/dli-slice.png) Open the file [exercises/example1_simple_slice.cu](exercises/example1_simple_slice.cu) and edit the contents where you see TODO markers.<jupyter_code>!./exercises/compile_and_run.sh example1_simple_slice<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000:    6    7 000001:   10   11 ``` The next line shows a variant of `Slice` that can reduce the dimension of a tensor:```c++auto t1 = t2.Slice({0, 1}, {matxEnd, matxDropDim});```Using this form of `Slice` requires a template argument with the rank of the new slice. The first parameter to `Slice` takes the starting index for each dimension, while the second takes the ending index. To include all values from the beginning on, a special sentinel of `matxEnd` can be used. Similarly, `matxDropDim` is used to indicate this dimension is the one being sliced (i.e. removed). In this case we are slicing the second column of the tensor and all rows, which produces a new 1D tensor containing only the second column of the original tensor. This is equivalent to `t2[:,1]` in Python. ![Column Slice](img/dli-slice_col.png)Open the file [exercises/example1_adv_slice_col.cu](exercises/example1_adv_slice_col.cu) and edit the contents where you see TODO markers.<jupyter_code>!./exercises/compile_and_run.sh example1_adv_slice_col<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000:    2 000001:    6 000002:   10 000003:   14 000004:   18 ``` Instead of slicing a single column, we can also slice a single row:```c++auto t1 = t2.Slice({1,0}, {matxDropDim, matxEnd});```![Row Slice](img/dli-slice_row.png)Open the file [exercises/example1_adv_slice_row.cu](exercises/example1_adv_slice_row.cu) and edit the contents where you see TODO markers.<jupyter_code>!./exercises/compile_and_run.sh example1_adv_slice_row<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000:    5 000001:    6 000002:    7 000003:    8 ``` Note that since we reduced the dimension to a 1D tensor in both cases, printing a 1D tensor (vector) will appear the same in the direction the values are printed. 5. CloneThe last line shows `Clone`, which replicates a tensor's dimensions into a higher-rank tensor:```c++auto t2c = t1.Clone({5, matxKeepDim});````Clone` is used on a 1D tensor from the output of the previous example, and replicates the data of the `t1` vector into a 2D tensor with 5 rows where all rows match the data in `t1`. Cloning does not replicate the data in memory; instead, the same elements in `t1` are accessed repeatedly when different rows are accessed. This not only saves memory, but also benefits from the caches in the GPU by not hitting different addresses in memory for the same value. In this case `Clone` was being used on a 1D view from a 2D tensor data set, but similar code works on taking any dimension tensor and increasing it to a higher dimension. The increase in dimensions is not restricted to one. For example, a scalar (0D tensor) can be cloned into a 4F tensor where a single value in memory would appear as a 4D tensor.![Permuted/Transposed 2D Tensor](img/dli-clone.png)Open the file [exercises/example1_clone.cu](exercises/example1_clone.cu) and edit the first TODO.<jupyter_code>!./exercises/compile_and_run.sh example1_clone<jupyter_output><empty_output><jupyter_text>Expected output:```sh000000: 1 2 3 4 000001: 1 2 3 4 000002: 1 2 3 4 000003: 1 2 3 4 000004: 1 2 3 4 ``` By changing which dimension is cloned, we can also take the same 1D tensor across columns. Edit the last file and clones across columns instead, and print the output of the cloned view.![Column Clone](img/dli-clone-col.png) Expected output:```sh000000: 1 1 1 1 1 000001: 2 2 2 2 2 000002: 3 3 3 3 3 000003: 4 4 4 4 4 ``` As mentioned at the beginning, views do not modify the underlying data; they simply provide the metadata needed to access the elements. To show this, we will open the same [exercise](exercises/example1_clone.cu) again and modify the first value in the original 1D tensor to 10 and watch how multiple elements of the cloned view are modified.<jupyter_code>!./exercises/compile_and_run.sh example1_clone<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0591884849236878, 'fraction_numerical': 0.028911775654547712, 'mean_word_length': 4.808648648648648, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '47758', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4503, 'n_tokens_neox': 4059, 'n_words': 2355, 'path': 'docs_input/notebooks/01_introduction.ipynb', 'repo_name': 'bonevbs/MatX', 'size': '16118'}