<commit_before>---
layout: post
title: How to use the ANU supercomputer
excerpt: "Everything I've learnt so far about how to use the NCI"
modified: 2014-10-24
tags: [NCI, ANU, Python]
comments: true
---

Here's what I've learnt so far trying to get my code running on ANU's [National Computational Infrastructure](http://nci.org.au). The NCI website provides a lot information on all of this (see the [raijin user guide](http://nci.org.au/services-support/getting-help/raijin-user-guide/)), but I've found it can be a bit opaque. Luckily the help desk staff are very responsive if you ever get stuck (<EMAIL>).

# What is the NCI?

The NCI houses raijin a 60,000 CPU core supercomputer - the largest in the Southern Hemisphere - and its located at the ANU next to South oval. 

<figure>
	<img src="http://nealbob.github.io/images/nci.jpg">
</figure>

# Applying for time on raijin

For ANU Phd students the first step is to apply for a startup grant. This is a small allocation of time (1000 core hours) to do do initial testing: learn how to use the system and give you an idea of how many core hours you will need for your main application. <br><br> To apply go [here](http://nci.org.au/access/user-registration/application-form-resource/), fill in the form and get your supervisor to sign (technically the application comes from them not you) then email it to the help desk (or just walk it over its not that far!). The application process was a bit confusing at times, but a few emails to the help desk got it done. <br><br> Full applications open in November for computing time the next year. The minimum request on raijin is 20,000 core hours, which is fairly large: equivalent to about 4000 hours or 20 weeks time on an i7 desktop or around $5000 worth on a commercial service like [multyvac](http://www.multyvac.com/). 

# Logging in

Once you're approved you'll receive a username and project code via email and a temporary password via sms.

You login online via ssh (if your on windows or mac you will need an ssh client, see the [user guide](http://nci.org.au/services-support/getting-help/raijin-user-guide/)), on linux just open the terminal and type 

{% highlight bash %}
ssh -l username raijin.nci.org.au
{% endhighlight %}

You'll get a warning about connecting for the first time, just type yes, then enter your password and your in.

{% highlight bash %}
###############################################################################
#                            NCI National Facility                            #
#      This service is for authorised clients only. It is a criminal          #
#      offence to:                                                            #
#                - Obtain access to data without permission                   #
#                - Damage, delete, alter or insert data without permission    #
#      Use of this system requires acceptance of the Conditions of Use        #
#      published at http://nci.org.au/conditions/                             #
###############################################################################
|                  Welcome to the NCI National Facility!                      |
|        raijin.nci.org.au - 57472 processor InfiniBand x86_64 cluster        |
|     Assistance: <EMAIL>     Information: http://nci.org.au          |
===============================================================================
{% endhighlight %}

# First steps

So now you are in a command line linux environment on a single raijin node (with 16 cores). You should have a prompt that looks like this

{% highlight bash %}
[username@raijin4 ~]$
{% endhighlight %}

You begin in your home folder, which is on the filesystem at /home/username. Here you can install your code and any dependencies. Any changes you make will be saved when you log out. Your home folder has a capacity of 2 Gb.

First lets reset our password

{% highlight bash %}
[username@raijin4 ~]$ passwd
{% endhighlight %}

Next we can check our account status

{% highlight bash %}
[username@raijin4 ~]$ nci_account

Usage Report: Project=fr3 Compute Period=2014.q4 (01/10/2014-31/12/2014)
========================================================================

Total Grant: 1000.00 SU
Total Used:   0.00 SU
Total Avail: 1000.00 SU
Bonus Used:   0.00 SU

-------------------------------------------------------------------------------------------------------------
System      Queue  Charge         Usage         Usage       SU used   Reserved SU    Pending SU      Total SU
                   Weight     (CPU Hrs)    (Walltime)                   (Running)      (Queued)     Committed
raijin      copyq     1.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin    express     3.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin    hugemem     1.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin     normal     1.0          0.00          0.00          0.00          0.00          0.00          0.00
-------------------------------------------------------------------------------------------------------------
Overall                            0.00          0.00          0.00          0.00          0.00          0.00


Usage Report: Project=fr3 Storage Period=2014.10 (01/10/2014-31/12/2014)
========================================================================
-------------------------------------------------------------------------------------------------
System    StoragePt             Grant       Usage       Avail      iGrant      iUsage      iAvail
-------------------------------------------------------------------------------------------------
dmf       massdata            20.00GB      0.00GB     20.00GB     100.00K       0.00K     100.00K
raijin    short               72.00GB      0.00GB     72.00GB     164.00K       0.01K     163.99K
-------------------------------------------------------------------------------------------------
Total                         92.00GB      0.00GB     92.00GB     264.00K       0.01K     263.99K
{% endhighlight %}

You should see your 1000 hour allocation. Note that these interactive sessions don't count towards your quota. 

# Installing software

raijin has just about all the software you might need already installed (see the [list](http://nci.org.au/nci-systems/national-facility/peak-system/raijin/application-software/)). You just need to make it available to your profile with the `module` command. To view the list of all software type

{% highlight bash %}
module avail
{% endhighlight %}

To install Python (with `numpy`, `scipy` and `matplotlib`) you type

{% highlight bash %}
module load python/2.7.3
module load python/2.7.3-matplotlib
{% endhighlight %}

To use Cython I also needed to replace the default intel C compiler with gcc

{% highlight bash %}
module unload intel-cc
module load gcc/4.9.0
{% endhighlight %}

You can then add these commands to your `.profile` file, to make sure they are executed on login. I used vim to edit these text files

{% highlight bash %}
vim .profile
{% endhighlight %}

# Installing your code and dependencies

An easy way to load your code is via [github](https://github.com) (which is like dropbox for code). Once you've learnt github, and have your code in a github repository [like this](https://github.com/nealbob/regrivermod), you can clone it directly onto your profile.

{% highlight bash %}
git clone git://github.com/nealbob/regrivermod.git ~/Model
{% endhighlight %}

Too easy. I also needed to compile my code, which once I installed `gcc` worked just like it does on my local machine. 

Next I needed to install a number of other Python packages not included on raijin by default (`cython`, `pandas`, `scikit-learn`). First create a folder to hold them

{% highlight bash %}
mkdir packages
{% endhighlight %}

For this I can use `easy_install`. You just need to tell `easy_install` to install locally, for example

{% highlight bash %}
easy_install --install-dir=~/packages pandas
{% endhighlight %}

Next you need to add ~/packages to your PYTHONPATH environment variable so Python can find it

{% highlight bash %}
export PYTHONPATH=~/packages:$PYTHONPATH
{% endhighlight %}

Its best to make this change permanent by adding it to your `.bashrc` file.

# Data storage

The first place to store large data files is in your 'short' folder, located at /short/project/username. As we saw from the `nci_account` output, you get 72 Gb on short and 20 on massdata. <br> <br> To transfer data between short on raijin to your local computer you use `rsync`. I don't have any large input files, but I do have to download some large result files, saved on to my short folder. To transfer a file from your `short` folder to you local machine, naviagate to the local folder you want to hold the file the type

rsync  -e "ssh -c arcfour" <EMAIL>:/short/projectcode/username/filename 

For longer term storage, the user guide recommends massdata because it is backed up.<commit_msg>Update 2014-10-24-usingtheNCI.md<commit_after>---
layout: post
title: How to use the ANU supercomputer
excerpt: "Everything I've learnt so far about how to use the NCI"
modified: 2014-10-24
tags: [NCI, ANU, Python]
comments: true
---

Here's what I've learnt so far trying to get my code running on ANU's [National Computational Infrastructure](http://nci.org.au). The NCI website provides a lot information on all of this (see the [raijin user guide](http://nci.org.au/services-support/getting-help/raijin-user-guide/)), but I've found it can be a bit opaque. Luckily the help desk staff are very responsive if you ever get stuck (<EMAIL>).

# What is the NCI?

The NCI houses raijin a 60,000 CPU core supercomputer - the largest in the Southern Hemisphere - and its located at the ANU next to South oval. 

<figure>
	<img src="http://nealbob.github.io/images/nci.jpg">
</figure>

# Applying for time on raijin

For ANU Phd students the first step is to apply for a startup grant. This is a small allocation of time (1000 core hours) to do do initial testing: learn how to use the system and give you an idea of how many core hours you will need for your main application. <br><br> To apply go [here](http://nci.org.au/access/user-registration/application-form-resource/), fill in the form and get your supervisor to sign (technically the application comes from them not you) then email it to the help desk (or just walk it over its not that far!). The application process was a bit confusing at times, but a few emails to the help desk got it done. <br><br> Full applications open in November for computing time the next year. The minimum request on raijin is 20,000 core hours, which is fairly large: equivalent to about 4000 hours or 20 weeks time on an i7 desktop or around $5000 worth on a commercial service like [multyvac](http://www.multyvac.com/). 

# Logging in

Once you're approved you'll receive a username and project code via email and a temporary password via sms.

You login online via ssh (if your on windows or mac you will need an ssh client, see the [user guide](http://nci.org.au/services-support/getting-help/raijin-user-guide/)), on linux just open the terminal and type 

{% highlight bash %}
ssh -l username raijin.nci.org.au
{% endhighlight %}

You'll get a warning about connecting for the first time, just type yes, then enter your password and your in.

{% highlight bash %}
###############################################################################
#                            NCI National Facility                            #
#      This service is for authorised clients only. It is a criminal          #
#      offence to:                                                            #
#                - Obtain access to data without permission                   #
#                - Damage, delete, alter or insert data without permission    #
#      Use of this system requires acceptance of the Conditions of Use        #
#      published at http://nci.org.au/conditions/                             #
###############################################################################
|                  Welcome to the NCI National Facility!                      |
|        raijin.nci.org.au - 57472 processor InfiniBand x86_64 cluster        |
|     Assistance: <EMAIL>     Information: http://nci.org.au          |
===============================================================================
{% endhighlight %}

# First steps

So now you are in a command line linux environment on a single raijin node (with 16 cores). You should have a prompt that looks like this

{% highlight bash %}
[username@raijin4 ~]$
{% endhighlight %}

You begin in your home folder, which is on the filesystem at /home/username. Here you can install your code and any dependencies. Any changes you make will be saved when you log out. Your home folder has a capacity of 2 Gb.

First lets reset our password

{% highlight bash %}
[username@raijin4 ~]$ passwd
{% endhighlight %}

Next we can check our account status

{% highlight bash %}
[username@raijin4 ~]$ nci_account

Usage Report: Project=fr3 Compute Period=2014.q4 (01/10/2014-31/12/2014)
========================================================================

Total Grant: 1000.00 SU
Total Used:   0.00 SU
Total Avail: 1000.00 SU
Bonus Used:   0.00 SU

-------------------------------------------------------------------------------------------------------------
System      Queue  Charge         Usage         Usage       SU used   Reserved SU    Pending SU      Total SU
                   Weight     (CPU Hrs)    (Walltime)                   (Running)      (Queued)     Committed
raijin      copyq     1.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin    express     3.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin    hugemem     1.0          0.00          0.00          0.00          0.00          0.00          0.00
raijin     normal     1.0          0.00          0.00          0.00          0.00          0.00          0.00
-------------------------------------------------------------------------------------------------------------
Overall                            0.00          0.00          0.00          0.00          0.00          0.00


Usage Report: Project=fr3 Storage Period=2014.10 (01/10/2014-31/12/2014)
========================================================================
-------------------------------------------------------------------------------------------------
System    StoragePt             Grant       Usage       Avail      iGrant      iUsage      iAvail
-------------------------------------------------------------------------------------------------
dmf       massdata            20.00GB      0.00GB     20.00GB     100.00K       0.00K     100.00K
raijin    short               72.00GB      0.00GB     72.00GB     164.00K       0.01K     163.99K
-------------------------------------------------------------------------------------------------
Total                         92.00GB      0.00GB     92.00GB     264.00K       0.01K     263.99K
{% endhighlight %}

You should see your 1000 hour allocation. Note that these interactive sessions don't count towards your quota. 

# Installing software

raijin has just about all the software you might need already installed (see the [list](http://nci.org.au/nci-systems/national-facility/peak-system/raijin/application-software/)). You just need to make it available to your profile with the `module` command. To view the list of all software type

{% highlight bash %}
module avail
{% endhighlight %}

To install Python (with `numpy`, `scipy` and `matplotlib`) you type

{% highlight bash %}
module load python/2.7.3
module load python/2.7.3-matplotlib
{% endhighlight %}

To use Cython I also needed to replace the default intel C compiler with gcc

{% highlight bash %}
module unload intel-cc
module load gcc/4.9.0
{% endhighlight %}

You can then add these commands to your `.profile` file, to make sure they are executed on login. I used vim to edit these text files

{% highlight bash %}
vim .profile
{% endhighlight %}

# Installing your code and dependencies

An easy way to load your code is via [github](https://github.com) (which is like dropbox for code). Once you've learnt github, and have your code in a github repository [like this](https://github.com/nealbob/regrivermod), you can clone it directly onto your profile.

{% highlight bash %}
git clone git://github.com/nealbob/regrivermod.git ~/Model
{% endhighlight %}

Too easy. I also needed to compile my code, which once I installed `gcc` worked just like it does on my local machine. 

Next I needed to install a number of other Python packages not included on raijin by default (`cython`, `pandas`, `scikit-learn`). First create a folder to hold them

{% highlight bash %}
mkdir packages
{% endhighlight %}

For this I can use `easy_install`. You just need to tell `easy_install` to install locally, for example

{% highlight bash %}
easy_install --install-dir=~/packages pandas
{% endhighlight %}

Next you need to add ~/packages to your PYTHONPATH environment variable so Python can find it

{% highlight bash %}
export PYTHONPATH=~/packages:$PYTHONPATH
{% endhighlight %}

Its best to make this change permanent by adding it to your `.bashrc` file.

# Data storage

The first place to store large data files is in your 'short' folder, located at /short/project/username. As we saw from the `nci_account` output, you get 72 Gb on short and 20 on massdata. <br> <br> To transfer data between short on raijin to your local computer you use `rsync`. I don't have any large input files, but I do have to download some large result files, saved on to my short folder. To transfer a file from your `short` folder to you local machine, naviagate to the local folder you want to hold the file the type

{% highlight bash %}
rsync  -e "ssh -c arcfour" <EMAIL>:/short/projectcode/username/filename 
{% endhighlight %}

For longer term storage, the user guide recommends massdata because it is backed up.
{'fraction_non_alphanumeric': 0.1657051810340091, 'fraction_numerical': 0.033184989835723315, 'mean_word_length': 2.188299176738483, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4869654', 'n_tokens_mistral': 5206, 'n_tokens_neox': 4524, 'n_words': 2119}