<commit_before>---
layout: post
title: Introduction to GPU programing through AMP C++
date: '2013-09-26T15:05:00.001-07:00'
author: <NAME>
tags:
- C++
- GPU
- Computer Science
modified_time: '2014-06-26T10:29:29.947-07:00'
thumbnail: http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s72-c/gpu_world.png
blogger_id: tag:blogger.com,1999:blog-1710034134179566048.post-1965487944885903339
blogger_orig_url: http://hoonzis.blogspot.com/2013/09/introduction-to-gpu-programing-through.html
---
Few months ago I tried to learn a bit about GPU programming and I took
notes a started to write this post. I am publishing this now even though
it's not complete, however being too busy, I am not sure whether I will
have the time to get back to this later.

Since couple years CUDA (\*2007) and OpenCL (\*2008) have established
themselves as standard frameworks for parallel programming on the GPU.
In 2011 new framework came to the landscape of GPGPU programming, which
does not compete on the same level, but rather represents new
abstraction between the GPU and the developer.

AMP stands for Accelerated Massive Parallelism and it is an open C++
standard. The implementation of AMP provided by Microsoft consists of
three main features:

-   C++ language features and compiler
-   Runtime
-   Programming framework containing classes and functions to facilitate
    parallel programming

As well as OpenCL and unlike CUDA, AMP is a standard not only for GPU
programming but for any data-parallel hardware. Typical example of such
a hardware is the vector unit of standard CPU, capable of executing SIMD
(Single Instruction Multiple Data) instructions. Or in the near future a
cloud based cluster. AMP is an abstraction layer which can be used to
target different devices from different vendors.

The following diagram depicts how AMP C++ fits to the world of GPGPU
programming.

[![](http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s400/gpu_world.png)](http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s1600/gpu_world.png)

Microsoft has implemented AMP on top of DirectCompute technology and it
is the only production ready implementation. Currently there are also
two Proof Of Concepts AMP implementations [Shelvin park
project](http://isocpp.org/blog/2012/12/shevlin-park) done by Intel and
[open source modification of the LLVM
compiler](https://github.com/corngood/llvm) ([more information
here](http://www.marshut.com/ipymi/c-amp-opencl-nvptx-prototype.html)).
Needless to say that the adoption of such standard will depend a lot on
whether these implementations will be taken further and whether we will
have real implementation based on CUDA.

First look at the code
----------------------

Without further introduction let's take a look at the classical "Hello
World" of GPGPU programming: Addition of elements of two vectors (here
without the boiler plate code).

```cpp
parallel_for_each(e,[=](index<2>idx) restrict(amp){
 c[idx] = a[idx] + b[idx];
});
```

If you have ever tried out OpenCL or CUDA, you should find this piece of
code less verbose and quite easy to read. That is the result of the main
ideas behind AMP C++:

-   **Full C++** - unlike CUDA or OpenCL, AMP is not a C like dialect.
-   **Minimalist API** - AMP hides as much as it can from the developer.
-   **Future proofed** - the API is designed for
    heterogeneous programming. In the future GPU should be only one of
    it's usages. It aims to be a standard for programming on
    distributed systems.

Before going futher with more examples we have to provide basic
description of GPU architecture as it is crutial to understanding
certain parts of AMP

GPU architecture
----------------

Here I will give a high level overview of the GPU architecture. If you
are already familiar with the GPU architecture, feel free to skip this
section.

Currently there are two main producers of GPU chips: NVidia and ATI
(AMD). A little behind comes Intel. Each of them constructs graphic
cards with different architecture. Moreover the architectures change
significantly with the realease of any new version. Nevertheless certain
concepts are shared by all of them. I will describe here shortly the
NVidias Fermi and ATIs Cypress architecture. The succesor of Fermi is
called
[Kepler](http://www.geforce.com/Active/en_US/en_US/pdf/GeForce-GTX-680-Whitepaper-FINAL.pdf).

### Processor architecture

The GPU is composed of hundreds of pipelined cores. The cores are
grouped to computation units. NVidia calls these units **Symmetric
Multiprocessors.** Each computation unit is assigned a unit of work.

[![](http://4.bp.blogspot.com/-aZCFecuHVPs/UkSvjHWnpJI/AAAAAAAAAsY/SbDYlzDG3Po/s400/fermi.png)](http://4.bp.blogspot.com/-aZCFecuHVPs/UkSvjHWnpJI/AAAAAAAAAsY/SbDYlzDG3Po/s1600/fermi.png)

ATI uses slightly different abstraction. Similary to NVidia the "cores"
are grouped to Symmetric Multiprocessors, however each core is called
**Thread Processor** and is a capable of executing VLIV (Very Long
Instruction Word) instructions. It has therefore 4 arithmetic logical
units and one Special Function Unit. The compiler has to find
independent operations and construct the VLIW instruction. If the
compiler is not able to find these independent operations than one or
more of the ALUs will not be performing any operation.

[![](http://2.bp.blogspot.com/-UtmWtZfmX6c/UkSvpeZehMI/AAAAAAAAAsg/ui4FBffCckA/s400/cypress.png)](http://2.bp.blogspot.com/-UtmWtZfmX6c/UkSvpeZehMI/AAAAAAAAAsg/ui4FBffCckA/s1600/cypress.png)

### Memory access

The GPU has a global memory which can be accessed by all cores. This
access is quite costly (hundreds of cycles). Each simetric
multiprocessor in turn has a small scratch-pad memory which is called
usually **local memory**. This memory can be accessed only by the
threads running on the given SM. The access to local memory is much
cheaper (around 10 cycles). This memory can either take a role of a
cache or can be managed by the developer directly. In addition each core
has its own general purpose registers, used to perform the computations.

GPUs usually run computations on data which is accessed only once,
unlike CPUs where the same memory pages are often used repetively. For
this reason it is better for the developer to control the local memory.
That's why historically the local memory did not act as cache on GPUs.
However this will probably change in the future, and some form of
caching will be provided by the GPU directly.

### Scheduling

Scheduling is essential because it allows the effective usage of the
processor. Since the memory access is expensive, other operations might
be executed while the processor is waiting for the data to come. NVidia
processors have "costless" context switching, so while one thread is
blocked other can take over. Therefore the number of threads scheduled
might affect the computation performance, while if not enough threads
are available for scheduling some might be waiting for the memory page
loads.

The programming models comparison:
----------------------------------

These architectural concepts are used by NVIDIA and ATI. Each of the 3
GPU programming models (CUDA, OpenCL, AMP) can has its own dialects and
namings. The following table shows the terms used by all three
technologies. We will discuss the AMP terms used in this table later in
the article.

|Term                                            |CUDA                    |AMP C++           |OpenCL      |
|------------------------------------------------|------------------------|------------------|------------|
|Basic unit of work                              |thread                  |thread            |work-item   |
|The code executed on one item                   |kernel                  |kernel            |kernel      |
|The unit processing one group of working units  |Streaming multiprocessor|-                 |Compute unit|
|Cache-style memory accessible by grouped threads|shared memory           |tile static memory|local memory|
|Group of working units sharing local memory     |warp                    |tile              |work-group  |

### The tools offered by each framework

Now when the architecture of GPU has been described we can define the
tools which each framework needs to provide. In the next part of the
article I will try to describe how AMP adresses these issues.

-   Tell the compiler which part of the code will be offloaded to the
    GPU
-   Provide constructs to work with multidimensional vectors
-   Provide a way to transfer the data between the GPU and the CPU
-   Give the developer tools to write efficient programs. That is to
    address GPU specific problems such as memory access

### AMP detailed description and more examples

Here is again the first example with the little of boiler plate code
that we have to write to make it work:

```cpp
void vectorsAddition_Parallel(vector<float> vA, vector<float> vB, vector<float>vC, int M, int N){
 extent<2> e(M,N);
 array_view<float,2> a(e, vA), b(e,vB);
 array_view<float,2> c(e, vC);<commit_msg>another try to get table right<commit_after>---
layout: post
title: Introduction to GPU programing through AMP C++
date: '2013-09-26T15:05:00.001-07:00'
author: <NAME>
tags:
- C++
- GPU
- Computer Science
modified_time: '2014-06-26T10:29:29.947-07:00'
thumbnail: http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s72-c/gpu_world.png
blogger_id: tag:blogger.com,1999:blog-1710034134179566048.post-1965487944885903339
blogger_orig_url: http://hoonzis.blogspot.com/2013/09/introduction-to-gpu-programing-through.html
---
Few months ago I tried to learn a bit about GPU programming and I took
notes a started to write this post. I am publishing this now even though
it's not complete, however being too busy, I am not sure whether I will
have the time to get back to this later.

Since couple years CUDA (\*2007) and OpenCL (\*2008) have established
themselves as standard frameworks for parallel programming on the GPU.
In 2011 new framework came to the landscape of GPGPU programming, which
does not compete on the same level, but rather represents new
abstraction between the GPU and the developer.

AMP stands for Accelerated Massive Parallelism and it is an open C++
standard. The implementation of AMP provided by Microsoft consists of
three main features:

-   C++ language features and compiler
-   Runtime
-   Programming framework containing classes and functions to facilitate
    parallel programming

As well as OpenCL and unlike CUDA, AMP is a standard not only for GPU
programming but for any data-parallel hardware. Typical example of such
a hardware is the vector unit of standard CPU, capable of executing SIMD
(Single Instruction Multiple Data) instructions. Or in the near future a
cloud based cluster. AMP is an abstraction layer which can be used to
target different devices from different vendors.

The following diagram depicts how AMP C++ fits to the world of GPGPU
programming.

[![](http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s400/gpu_world.png)](http://3.bp.blogspot.com/-XNWrfsNIS7I/UkSvbyVrjYI/AAAAAAAAAsQ/UyOQkpJTkl8/s1600/gpu_world.png)

Microsoft has implemented AMP on top of DirectCompute technology and it
is the only production ready implementation. Currently there are also
two Proof Of Concepts AMP implementations [Shelvin park
project](http://isocpp.org/blog/2012/12/shevlin-park) done by Intel and
[open source modification of the LLVM
compiler](https://github.com/corngood/llvm) ([more information
here](http://www.marshut.com/ipymi/c-amp-opencl-nvptx-prototype.html)).
Needless to say that the adoption of such standard will depend a lot on
whether these implementations will be taken further and whether we will
have real implementation based on CUDA.

First look at the code
----------------------

Without further introduction let's take a look at the classical "Hello
World" of GPGPU programming: Addition of elements of two vectors (here
without the boiler plate code).

```cpp
parallel_for_each(e,[=](index<2>idx) restrict(amp){
 c[idx] = a[idx] + b[idx];
});
```

If you have ever tried out OpenCL or CUDA, you should find this piece of
code less verbose and quite easy to read. That is the result of the main
ideas behind AMP C++:

-   **Full C++** - unlike CUDA or OpenCL, AMP is not a C like dialect.
-   **Minimalist API** - AMP hides as much as it can from the developer.
-   **Future proofed** - the API is designed for
    heterogeneous programming. In the future GPU should be only one of
    it's usages. It aims to be a standard for programming on
    distributed systems.

Before going futher with more examples we have to provide basic
description of GPU architecture as it is crutial to understanding
certain parts of AMP

GPU architecture
----------------

Here I will give a high level overview of the GPU architecture. If you
are already familiar with the GPU architecture, feel free to skip this
section.

Currently there are two main producers of GPU chips: NVidia and ATI
(AMD). A little behind comes Intel. Each of them constructs graphic
cards with different architecture. Moreover the architectures change
significantly with the realease of any new version. Nevertheless certain
concepts are shared by all of them. I will describe here shortly the
NVidias Fermi and ATIs Cypress architecture. The succesor of Fermi is
called
[Kepler](http://www.geforce.com/Active/en_US/en_US/pdf/GeForce-GTX-680-Whitepaper-FINAL.pdf).

### Processor architecture

The GPU is composed of hundreds of pipelined cores. The cores are
grouped to computation units. NVidia calls these units **Symmetric
Multiprocessors.** Each computation unit is assigned a unit of work.

[![](http://4.bp.blogspot.com/-aZCFecuHVPs/UkSvjHWnpJI/AAAAAAAAAsY/SbDYlzDG3Po/s400/fermi.png)](http://4.bp.blogspot.com/-aZCFecuHVPs/UkSvjHWnpJI/AAAAAAAAAsY/SbDYlzDG3Po/s1600/fermi.png)

ATI uses slightly different abstraction. Similary to NVidia the "cores"
are grouped to Symmetric Multiprocessors, however each core is called
**Thread Processor** and is a capable of executing VLIV (Very Long
Instruction Word) instructions. It has therefore 4 arithmetic logical
units and one Special Function Unit. The compiler has to find
independent operations and construct the VLIW instruction. If the
compiler is not able to find these independent operations than one or
more of the ALUs will not be performing any operation.

[![](http://2.bp.blogspot.com/-UtmWtZfmX6c/UkSvpeZehMI/AAAAAAAAAsg/ui4FBffCckA/s400/cypress.png)](http://2.bp.blogspot.com/-UtmWtZfmX6c/UkSvpeZehMI/AAAAAAAAAsg/ui4FBffCckA/s1600/cypress.png)

### Memory access

The GPU has a global memory which can be accessed by all cores. This
access is quite costly (hundreds of cycles). Each simetric
multiprocessor in turn has a small scratch-pad memory which is called
usually **local memory**. This memory can be accessed only by the
threads running on the given SM. The access to local memory is much
cheaper (around 10 cycles). This memory can either take a role of a
cache or can be managed by the developer directly. In addition each core
has its own general purpose registers, used to perform the computations.

GPUs usually run computations on data which is accessed only once,
unlike CPUs where the same memory pages are often used repetively. For
this reason it is better for the developer to control the local memory.
That's why historically the local memory did not act as cache on GPUs.
However this will probably change in the future, and some form of
caching will be provided by the GPU directly.

### Scheduling

Scheduling is essential because it allows the effective usage of the
processor. Since the memory access is expensive, other operations might
be executed while the processor is waiting for the data to come. NVidia
processors have "costless" context switching, so while one thread is
blocked other can take over. Therefore the number of threads scheduled
might affect the computation performance, while if not enough threads
are available for scheduling some might be waiting for the memory page
loads.

The programming models comparison:
----------------------------------

These architectural concepts are used by NVIDIA and ATI. Each of the 3
GPU programming models (CUDA, OpenCL, AMP) can has its own dialects and
namings. The following table shows the terms used by all three
technologies. We will discuss the AMP terms used in this table later in
the article.

| Term                                             | CUDA                     | AMP C++            | OpenCL       |
| ------------------------------------------------ | ------------------------ | ------------------ | ------------ |
| Basic unit of work                               | thread                   | thread             | work-item    |
| The code executed on one item                    | kernel                   | kernel             | kernel       |
| The unit processing one group of working units   | Streaming multiprocessor | -                  | Compute unit |
| Cache-style memory accessible by grouped threads | shared memory            | tile static memory | local memory |
| Group of working units sharing local memory      | warp                     | tile               | work-group   |

### The tools offered by each framework

Now when the architecture of GPU has been described we can define the
tools which each framework needs to provide. In the next part of the
article I will try to describe how AMP adresses these issues.

-   Tell the compiler which part of the code will be offloaded to the
    GPU
-   Provide constructs to work with multidimensional vectors
-   Provide a way to transfer the data between the GPU and the CPU
-   Give the developer tools to write efficient programs. That is to
    address GPU specific problems such as memory access

### AMP detailed description and more examples

Here is again the first example with the little of boiler plate code
that we have to write to make it work:

```cpp
void vectorsAddition_Parallel(vector<float> vA, vector<float> vB, vector<float>vC, int M, int N){
 extent<2> e(M,N);
 array_view<float,2> a(e, vA), b(e,vB);
 array_view<float,2> c(e, vC);
{'fraction_non_alphanumeric': 0.07840989979740459, 'fraction_numerical': 0.01763127635109237, 'mean_word_length': 3.5455450472872077, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 14, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4246797', 'n_tokens_mistral': 5275, 'n_tokens_neox': 4816, 'n_words': 2392}