Introducing HPC with a Raspberry Pi cluster
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Sauze, Colin
This talk will discuss the development of a RaspberryPi cluster for teaching an introduction to HPC. The motivation for this was to overcome four key problems faced by new HPC users: 1. The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course. 2. A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system. 3. That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using. 4. That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this. The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered.
 Just in time for the first talk, we have Colin who will be talking about building clusters with Raspberry Pi's. Should be green. Is that working? Yes. Right. Hello. My name is Colin Soze. I'm a research software engineer at Aberystwyth University in Wales. I'm going to talk today about my experience of building Raspberry Pi clusters and using them to teach people beginning their first journey into high-performance computing. Just a bit about me. I'm working on a project called Supercomputing Wales. We work across four universities in Wales, ourselves, Bangor University in the north and Cardiff and Swansea in the south. In this, we share two physical high-performance computing systems, one in Swansea and one in Cardiff. Across the project, we've employed 15 research software engineers such as myself whose role is not just to optimise software for any users but also to enable training and to help people become familiar with using the system. My background is actually a bit different to this. I did a PhD in robotics. I spent a lot of time playing with single-board Linux computers and various other embedded Linux devices. The Raspberry Pi came out just after I finished that, and I converted to it very quickly because very suddenly I realised that the computers I had been paying $200 or $300 a board for were now $20 or $30 with pretty much the same capabilities. I read a lot in the early days about people building Raspberry Pi clusters, and I thought, that sounds fun, but I can't see why you do it, apart from for the sake of doing it. The performance of a Raspberry Pi is so small that any real computation you want to run is better off done on your desktop computer than it is done on a Raspberry Pi cluster. Then more recently, I started to see people doing slightly more serious things, that Los Alamos National Laboratory in America built a 750 node Raspberry Pi cluster in this giant, I think it's a 4U rack mount case, and they actually use it as a test system for anyone who wants to run software on their real cluster to take the load off the real system. You can run a test case, do it on the Raspberry Pi, you prove it works on there, then you're allowed access to the real system. At the opposite end of the scale, EPCC in Edinburgh in Scotland built a system called We Archie named after their Archer supercomputer, which is the largest public supercomputer in the UK. They've also got an even smaller version that they call We Arch Let, which is what you see in this picture, which is just five Raspberry Pis in Lego cases. They take these out for hundreds of outreach demos to schools, science events, all sorts. Very helpfully, they put some instructions on their website about how to build one. This is what I started following as the basis for mine, but I found these were bit out of date. Also, they didn't go anything beyond doing basic MPI run. There was no job scheduling system, there was no real software environment, there was just basic MPI and a few demos. Since I first looked at it, though, they've actually come back and built about 20 or 30 really good demo programs. They're something I'd like to re-evaluate in a bit. My final inspiration for this was my colleagues in Swansea University actually got some money together to build a 16 node cluster that they demoed at the Swansea Festival of Science in 2018. They had a demonstration running with CFD, Computational Fluid Dynamics, using a Microsoft Connect sensor, modeling the airflow of how air would move over a person. They basically took a 3D picture of a person and then tried to show how the airflow would move around them. They've done a lot of work with this thing in real research. They are behind the CFD modeling for Bloodhound, the supersonic car that's currently in South Africa attempting some record breaking high-speed runs. Why should we actually teach with Raspberry Pi cluster instead of a real system? The first thing is, the real system is busy doing research. It's often that we see 85, 90, 95% load on our real systems. The last thing you actually want is a class of students coming in and using it and disrupting the real research. There's also, I found a fear amongst learners that they are going to break it. They are using this multi-million pound research equipment for the first time. A lot of the people coming into it for the first time don't have huge amounts of computational background. It's a bit scary suddenly being told, here go and play with this very expensive system. Whereas being told, here go and play with this very cheap system that's right in front of you seems a bit less daunting to them. Also, when you make mistakes with resource allocation of a big system because it's so big and you're doing sort of simple demonstration programs, it's not always immediately obvious that you've done something wrong. When you've got a much smaller system with much less memory, much less disk, much less compute, it becomes much more obvious, much more quickly that you've actually made a mistake in what you're doing. It's also nice to have control over our environment. On the real system, we are at the behest of our system administrators and to some extent our suppliers as to what we can change and how things go. So if we need a particular piece of software installing for a lesson, it might take a few days to get it installed and we've got a way to another people to install it. So the hardware is abstract. It's a way in a remote data center. For us, it's even in another university that's two hours drive away. Even for the people in that university, they're never going to see the real system. It's hidden in the basement somewhere that they are never going to see. Having the real system there in front of them makes it a lot less abstract, even if it doesn't quite look the same. Finally is that they don't need accounts. It does take us a few days to get people accounts. They need to already be registered on our system as students or staff and that can cause problems. You always get one person who shows up in a lesson who hasn't got their account ready. You go and create an account there and then and it's at least an hour until they can log on and do anything. Having a Raspberry Pi system there in front of them means we've got something we can immediately make an account for. No one cares that we made an account for them on it and they can log straight in. Also added to that is that the Raspberry Pi cluster presents as its own Wi-Fi access point. So anybody with a laptop can connect straight to it without having to get on the rest of the university network. So my little cluster I've called TweetyPi, following a theme of naming all of our clusters after birds. This was literally begged, borrowed and stolen from parts I found lying around the university. So it is 10 of the oldest original Raspberry Pi Model 1s. One Raspberry Pi 3, which is I think a 2017-18 model, to act as the head node and the login node. Running the latest version of Razbian, which is the Debian derivative for the Raspberry Pi, and the head node acts as Wi-Fi access point. The whole thing actually needs internet access to work because you have to have a synchronized clock across all the systems. The Raspberry Pi has no built-in real-time clock module. The easiest way to get that is to get up the internet and pull it off an NTP server. For that I then plug in a laptop, which is on the main Wi-Fi or a mobile phone, and then run the entire system through that. Demo software is something that's still a bit early days for. I very quickly wrote a demo program for National Science Week in March last year, which was simulating, calculating Pi using a Monte Carlo method. So here you put a load of random dots on a square and draw a quarter of a circle. Those that fall inside the circle are counted, and those that fall outside are the ratio. Between them gives you an approximation of Pi. The more points you use, the better your approximation gets, up to probably 15,000-20,000 points. The version I've written this is MPI, so you can choose to distribute it across all the compute nodes, and you can choose how many compute nodes you'd like to use. There's a very simple graphical interface to it that lets the user choose how many points they want to calculate and how many nodes they want to use. And then a very simple illustration of the queue. So they get to see how job queuing works, so one person can submit something to all 10 nodes, but the next person can't run anything. But obviously those 10 compute faster than if you just submitted it to three or four nodes. I mentioned earlier that my colleagues in Swansea have done a more impressive graphical demo using CFD. It's a bit more power hungry, though. They've got 16 Raspberry Pi 3s, which are quad core. I've got 10 Raspberry Pi 1s, which are single core and a much older processor design. I probably wouldn't be able to run their thing in a sensible amount of time, and even on their cluster it's taking three or four minutes per person to run. If you're at a science event trying to pull through many people in one go, that doesn't really work very well. Only in the recent months did I come across the demos from EPCC with the Wearchee and Archlet demos. I think there are 10 or 20 demos on their website, and that's certainly one of my things to evaluate for the future. I wanted to make the environment as realistic as possible. Most of the cluster designs I'd seen out there didn't do this. They went for basic running MPI, and that was it. I went for first installing MPI CH, and then I installed Slurm Job Manager so you can run all the standard Slurm commands, and all our main real clusters run Slurm as well. I put quotas on home directories to stop people creating too much data on there. All home directories are NFS mounted across the entire cluster. There are software modules through the TCL modules command, of which there's only one at the moment, which is a version of Python that's not available through the standard repositories. All the nodes network boot off one image, so I don't have to go and replicate SD cards with gigabytes of data and keep updating them every time I install a new piece of software. That causes a slight problem. The Raspberry Pi 1 didn't support network booting. It only came in in the Raspberry Pi 3. There's a slight hack to this. I placed the kernel and the boot loader on the SD card, and then I NFS mount the root file system. Now there's about a 50 megabyte image, which is identical for every machine on the SD card. That image doesn't update very often, only when there's a kernel update. All the other software is then pulled over the network via NFS. If I ever do get Raspberry Pi 3s, then moving this up to PXE booting on the Raspberry Pi 3 should be a trivial change of basically copying that SD card image to a TFTP boot server. For teaching materials, I had already written a lesson based around the principles of software carpentry. I don't know how many people here are familiar with software carpentry, but okay, one, two. An organization who goes out to build computational training for researchers, and they focus very much on trying to teach enough skills to be useful, not trying to teach an entire theoretical background and the whole of computer science. I'd already written a lesson loosely based on another lesson called HPC in a Day, which we talk about what is an HPC, how you log into it, what file system is available and how you can transfer data, submitting jobs and monitoring jobs at SLURM. Some more advanced stuff on how to do parallel processing, so a bit on profiling and measuring where performance bottlenecks exist, parallelizing code and looking at Amdahl's law for limitations, then taking that parallel code and moving it to MPI, and finally a bit on HPC best practice. What I did was I adapted this lesson to be entirely on the Raspberry Pi cluster. I removed any bits that didn't work for the Raspberry Pi, and it probably shrunk down to what should have been a three hour, three and a half hour lesson. I was then asked if I could teach something for a bunch of summer school students who were coming to visit us, and this unfortunately got cut down to about an hour and a half, so I ran a very rapid version of this. These PhD students were new students doing solar physics, all funded by STFC, the UK Science Technology Facilities Council. Every PhD student who was starting a PhD in solar physics that year came to us. One feature of this was none of them had actually registered for their courses yet. They were all just about to start, so none of them actually had a university account, which meant getting them onto any real HPC would have been an utter pain. Having something there on a desk that they could all just connect to from their laptops certainly made life a lot easier for that. For most of them it was the first time that they'd ever used an HPC system, but most of them had some unix experience already. They seemed to really enjoy it. They seemed to really enjoy actually playing around with just SSH into a system, and the idea that they were sharing a system over SSH, I don't know what it was, but they really really took that idea, but also the kind of idea that they didn't care about the system, they couldn't break it, but also that they could kind of mess around and see what each other was up to and see what was in each other's home directories, because I hadn't set permissions to be world readable. That seemed to really grab them for some reason. Unfortunately, the main complaint with that was just lack of time, so we ran through the whole thing in one and a half hours, and we got as far as just doing basic job submission. But one thing that was really obvious is with 15 people and a 10 node cluster, is queuing became really apparent really easily, because one person would submit a five or a 10 node job and instantly the queue is full. What I did was actually left up on the projector the output of the watch SQ command, so you saw every second an update of the queue, and you could see who was in the queue, and they could go and blame each other and see how long jobs were taking. A couple of issues we were just running that is I accidentally overwrote subsystem accounts when I created all the user accounts, and Slurm wouldn't run correctly on the first day, and that all the Wi-Fi was going through one laptop that was on actually our guest Wi-Fi because I hadn't registered the laptop properly, and that caused a few internet problems. Because they were all connecting to the cluster with their laptops for their Wi-Fi, that meant that all of their internet traffic was also going through that one laptop, and that was where they were getting the course notes from. And it meant that they were often flipping backwards and forwards between other networks or their phones and that main Wi-Fi for the cluster. I then ran it again for a second time. This time with a slightly different group, this was approximately 10 people who were actually doing proper training on how to use our real research cluster. It was about 10 people with a mix of levels, a mix of staff and students, and this time I simultaneously ran the real HPC alongside the Raspberry Pi cluster, so people could choose to use one or the other or both. It proved to be a really good backup for those who had locked their accounts, of which I think there were two. It helped them make it a bit more tangible, but a lot tended to pick one system or the other and would submit all their commands on one or the other, but I managed to get the environments close enough that it didn't matter. They were able to run everything that was in the material on either system. So in future work, what I would like to do is have a bit better configuration management for this. Right now I do have a set of scripts that should theoretically deploy a cluster. Testing these takes ages because you've got to start from the beginning and build software and go through the whole deployment process. I think on the Raspberry Pi ones, compiling the software that I've put in just for that one module takes about a day. So it's not a quick process to iterate through the development of this. The other big difference at the moment between this and any real system is this runs on Raspbian, which is a Debian derivative. Pretty much every HPC system I've ever come across runs on some kind of CentOS or Red Hat Enterprise Linux derivative. And it would be nice to try and get CentOS to run on there instead of Debian. As far as I understand, there is an ARM port that will run on the Raspberry Pi, but I haven't run to trying it yet, and Raspberry Pi is kind of the default Raspberry Pi foundation push. And I'd also like to focus a bit better on some demos. The tool I really want to do is we have a large group who analyze satellite imagery. I'd like to write something around there where someone can type in an address and then see some satellite imagery of that address and then do some analysis of it to look at how that land use in that area has changed over time. The other big area of research for us is in genomics, and we had two PhD students a few years ago develop something called Monster Lab. And here you put a set of LEGO bricks together, and the different colors of the LEGO bricks determine what gene you're looking at. You run this through a sequencer, which looks at the colors of the bricks, and then determines what the genetic sequence is and therefore what attributes your monster has. So for instance, the first LEGO brick is the number of heads, the second one is the number of legs, the third one is the number of arms, the fourth one is the number of eyes. And by having different colors, you then get different attributes to your monster. Right now, all the system does is you read through the brick and it tells you what the attributes are and the kids then go and draw their monsters. What I'd really like to do is simulate them and have maybe some kind of physics simulation that goes and simulates them. Or to do some gene sequence pattern searching and look for similar monsters that came up previously, which I suppose is the more realistic scenario to real genomics. So finally, if you're interested in building your own cluster, I have put up on my GitHub all of the scripts that I have used to create this. These are still under development somewhat. I've basically done one full iteration through them. Copying the code to the network booting is not fully tested, but you should at least be able to get a SD card deployed for the head node and for a compute node. Also up on GitHub, this is actually under my organizational one is the teaching material that I used. If you want to know any more about this, then please feel free to email me. My email address is there on the bottom of the slide. Do you want to have a question? James, please thank the speaker. Before you leave, please keep seated while we do questions. Any questions for Colin? Thank you for a very interesting presentation. I was wondering about support for containers. I see that coming more and more with singularity. Could that be added? I don't know if singularity works on ARM. I know Docker definitely doesn't, but I'm not sure if singularity can make work on an ARM system. Given that we're seeing some real ARM clusters, in the UK there's one called Ismbard at Bristol University. That's something that's in the pipework for singularity. It's definitely a feature that would be nice to add. Thank you. Any other questions? Please stay seated while we have another question. Is it essential to boot a Raspberry Pi from an image rather than install it in a conventional way? Is there any news on an image for CentOS for Raspberry Pi? You put an image on an SD card. You have to have an SD card in the older pies to boot from. My problem is if I want to put 10 images that are identical onto 10 machines, I've got to get the image right, then copy it. Then if I make a change, I've got to either redo those changes to every machine or rebuild the SD card image and then put a new SD card in every one of them. So by having a network boot, that problem goes away. Sorry, what was the second bit about CentOS? Is there any news on a CentOS image for the Raspberry Pi? I think there is a CentOS image, but I'm not certain. Right. Thanks. There's definitely an ARM image. In the back, please. Work together. I know you can. Thanks. Was it a Vessier first choice, or did you also try other file systems in networking systems like Lustre, GFS, whatever? I think Lustre might be a bit too much for a Raspberry Pi. Given the size of the cluster, just NFS is fine. There's not a performance issue, really. I'd also need multiple disks if I wanted to use a parallel file system. Having a single disk being served over a parallel file system won't achieve anything. I'm not fully aware of HPCs that much, so it might be a newbie question. But as a user, the PhDs who are using, like they're logging on onto the Raspberry Pi, do they see any difference instead of just SSH into a just regular node instead of HPC? Do they experience any different? Not really. That's part of the point, is that the experience is very similar to the real system. The main difference is obviously that it's got less memory, less disk space, and it's slower. But that's not immediately obvious when you're just typing commands at an SSH prompt. So what would be an advantage for them to train on?
 Just in time for the first talk, we have Colin who will be talking about building clusters  with Raspberry Pi's.  Should be green.  Is that working?  Yes.  Right.  Hello.  My name is Colin Soze.  I'm a research software engineer at Aberystwyth University in Wales.  I'm going to talk today about my experience of building Raspberry Pi clusters and using  them to teach people beginning their first journey into high-performance computing.  Just a bit about me.  I'm working on a project called Supercomputing Wales.  We work across four universities in Wales, ourselves, Bangor University in the north  and Cardiff and Swansea in the south.  In this, we share two physical high-performance computing systems, one in Swansea and one  in Cardiff.  Across the project, we've employed 15 research software engineers such as myself whose role  is not just to optimise software for any users but also to enable training and to help  people become familiar with using the system.  My background is actually a bit different to this.  I did a PhD in robotics.  I spent a lot of time playing with single-board Linux computers and various other embedded  Linux devices.  The Raspberry Pi came out just after I finished that, and I converted to it very quickly because  very suddenly I realised that the computers I had been paying $200 or $300 a board for  were now $20 or $30 with pretty much the same capabilities.  I read a lot in the early days about people building Raspberry Pi clusters, and I thought,  that sounds fun, but I can't see why you do it, apart from for the sake of doing it.  The performance of a Raspberry Pi is so small that any real computation you want to run  is better off done on your desktop computer than it is done on a Raspberry Pi cluster.  Then more recently, I started to see people doing slightly more serious things, that Los  Alamos National Laboratory in America built a 750 node Raspberry Pi cluster in this giant,  I think it's a 4U rack mount case, and they actually use it as a test system for anyone  who wants to run software on their real cluster to take the load off the real system.  You can run a test case, do it on the Raspberry Pi, you prove it works on there, then you're  allowed access to the real system.  At the opposite end of the scale, EPCC in Edinburgh in Scotland built a system called  We Archie named after their Archer supercomputer, which is the largest public supercomputer  in the UK.  They've also got an even smaller version that they call We Arch Let, which is what you see  in this picture, which is just five Raspberry Pis in Lego cases.  They take these out for hundreds of outreach demos to schools, science events, all sorts.  Very helpfully, they put some instructions on their website about how to build one.  This is what I started following as the basis for mine, but I found these were bit out of  date.  Also, they didn't go anything beyond doing basic MPI run.  There was no job scheduling system, there was no real software environment, there was  just basic MPI and a few demos.  Since I first looked at it, though, they've actually come back and built about 20 or 30  really good demo programs.  They're something I'd like to re-evaluate in a bit.  My final inspiration for this was my colleagues in Swansea University actually got some money  together to build a 16 node cluster that they demoed at the Swansea Festival of Science  in 2018.  They had a demonstration running with CFD, Computational Fluid Dynamics, using a Microsoft  Connect sensor, modeling the airflow of how air would move over a person.  They basically took a 3D picture of a person and then tried to show how the airflow would  move around them.  They've done a lot of work with this thing in real research.  They are behind the CFD modeling for Bloodhound, the supersonic car that's currently in South  Africa attempting some record breaking high-speed runs.  Why should we actually teach with Raspberry Pi cluster instead of a real system?  The first thing is, the real system is busy doing research.  It's often that we see 85, 90, 95% load on our real systems.  The last thing you actually want is a class of students coming in and using it and disrupting  the real research.  There's also, I found a fear amongst learners that they are going to break it.  They are using this multi-million pound research equipment for the first time.  A lot of the people coming into it for the first time don't have huge amounts of computational  background.  It's a bit scary suddenly being told, here go and play with this very expensive system.  Whereas being told, here go and play with this very cheap system that's right in front  of you seems a bit less daunting to them.  Also, when you make mistakes with resource allocation of a big system because it's so  big and you're doing sort of simple demonstration programs, it's not always immediately obvious  that you've done something wrong.  When you've got a much smaller system with much less memory, much less disk, much less  compute, it becomes much more obvious, much more quickly that you've actually made a mistake  in what you're doing.  It's also nice to have control over our environment.  On the real system, we are at the behest of our system administrators and to some extent  our suppliers as to what we can change and how things go.  So if we need a particular piece of software installing for a lesson, it might take a few  days to get it installed and we've got a way to another people to install it.  So the hardware is abstract.  It's a way in a remote data center.  For us, it's even in another university that's two hours drive away.  Even for the people in that university, they're never going to see the real system.  It's hidden in the basement somewhere that they are never going to see.  Having the real system there in front of them makes it a lot less abstract, even if it doesn't  quite look the same.  Finally is that they don't need accounts.  It does take us a few days to get people accounts.  They need to already be registered on our system as students or staff and that can cause  problems.  You always get one person who shows up in a lesson who hasn't got their account ready.  You go and create an account there and then and it's at least an hour until they can log  on and do anything.  Having a Raspberry Pi system there in front of them means we've got something we can immediately  make an account for.  No one cares that we made an account for them on it and they can log straight in.  Also added to that is that the Raspberry Pi cluster presents as its own Wi-Fi access point.  So anybody with a laptop can connect straight to it without having to get on the rest of  the university network.  So my little cluster I've called TweetyPi, following a theme of naming all of our clusters  after birds.  This was literally begged, borrowed and stolen from parts I found lying around the university.  So it is 10 of the oldest original Raspberry Pi Model 1s.  One Raspberry Pi 3, which is I think a 2017-18 model, to act as the head node and the login  node.  Running the latest version of Razbian, which is the Debian derivative for the Raspberry Pi,  and the head node acts as Wi-Fi access point.  The whole thing actually needs internet access to work because you have to have a synchronized  clock across all the systems.  The Raspberry Pi has no built-in real-time clock module.  The easiest way to get that is to get up the internet and pull it off an NTP server.  For that I then plug in a laptop, which is on the main Wi-Fi or a mobile phone, and then  run the entire system through that.  Demo software is something that's still a bit early days for.  I very quickly wrote a demo program for National Science Week in March last year, which was  simulating, calculating Pi using a Monte Carlo method.  So here you put a load of random dots on a square and draw a quarter of a circle.  Those that fall inside the circle are counted, and those that fall outside are the ratio.  Between them gives you an approximation of Pi.  The more points you use, the better your approximation gets, up to probably 15,000-20,000 points.  The version I've written this is MPI, so you can choose to distribute it across all the  compute nodes, and you can choose how many compute nodes you'd like to use.  There's a very simple graphical interface to it that lets the user choose how many  points they want to calculate and how many nodes they want to use.  And then a very simple illustration of the queue.  So they get to see how job queuing works, so one person can submit something to all  10 nodes, but the next person can't run anything.  But obviously those 10 compute faster than if you just submitted it to three or four  nodes.  I mentioned earlier that my colleagues in Swansea have done a more impressive graphical demo  using CFD.  It's a bit more power hungry, though.  They've got 16 Raspberry Pi 3s, which are quad core.  I've got 10 Raspberry Pi 1s, which are single core and a much older processor design.  I probably wouldn't be able to run their thing in a sensible amount of time, and even on  their cluster it's taking three or four minutes per person to run.  If you're at a science event trying to pull through many people in one go, that doesn't  really work very well.  Only in the recent months did I come across the demos from EPCC with the Wearchee and  Archlet demos.  I think there are 10 or 20 demos on their website, and that's certainly one of my things  to evaluate for the future.  I wanted to make the environment as realistic as possible.  Most of the cluster designs I'd seen out there didn't do this.  They went for basic running MPI, and that was it.  I went for first installing MPI CH, and then I installed Slurm Job Manager so you can run  all the standard Slurm commands, and all our main real clusters run Slurm as well.  I put quotas on home directories to stop people creating too much data on there.  All home directories are NFS mounted across the entire cluster.  There are software modules through the TCL modules command, of which there's only one  at the moment, which is a version of Python that's not available through the standard  repositories.  All the nodes network boot off one image, so I don't have to go and replicate SD cards  with gigabytes of data and keep updating them every time I install a new piece of software.  That causes a slight problem.  The Raspberry Pi 1 didn't support network booting.  It only came in in the Raspberry Pi 3.  There's a slight hack to this.  I placed the kernel and the boot loader on the SD card, and then I NFS mount the root  file system.  Now there's about a 50 megabyte image, which is identical for every machine on the SD  card.  That image doesn't update very often, only when there's a kernel update.  All the other software is then pulled over the network via NFS.  If I ever do get Raspberry Pi 3s, then moving this up to PXE booting on the Raspberry Pi  3 should be a trivial change of basically copying that SD card image to a TFTP boot  server.  For teaching materials, I had already written a lesson based around the principles of software  carpentry.  I don't know how many people here are familiar with software carpentry, but okay, one, two.  An organization who goes out to build computational training for researchers, and they focus very  much on trying to teach enough skills to be useful, not trying to teach an entire theoretical  background and the whole of computer science.  I'd already written a lesson loosely based on another lesson called HPC in a Day, which  we talk about what is an HPC, how you log into it, what file system is available and  how you can transfer data, submitting jobs and monitoring jobs at SLURM.  Some more advanced stuff on how to do parallel processing, so a bit on profiling and measuring  where performance bottlenecks exist, parallelizing code and looking at Amdahl's law for limitations,  then taking that parallel code and moving it to MPI, and finally a bit on HPC best practice.  What I did was I adapted this lesson to be entirely on the Raspberry Pi cluster.  I removed any bits that didn't work for the Raspberry Pi, and it probably shrunk down  to what should have been a three hour, three and a half hour lesson.  I was then asked if I could teach something for a bunch of summer school students who  were coming to visit us, and this unfortunately got cut down to about an hour and a half,  so I ran a very rapid version of this.  These PhD students were new students doing solar physics, all funded by STFC, the UK  Science Technology Facilities Council.  Every PhD student who was starting a PhD in solar physics that year came to us.  One feature of this was none of them had actually registered for their courses yet.  They were all just about to start, so none of them actually had a university account,  which meant getting them onto any real HPC would have been an utter pain.  Having something there on a desk that they could all just connect to from their laptops  certainly made life a lot easier for that.  For most of them it was the first time that they'd ever used an HPC system, but most of  them had some unix experience already.  They seemed to really enjoy it.  They seemed to really enjoy actually playing around with just SSH into a system, and the  idea that they were sharing a system over SSH, I don't know what it was, but they really  really took that idea, but also the kind of idea that they didn't care about the system,  they couldn't break it, but also that they could kind of mess around and see what each  other was up to and see what was in each other's home directories, because I hadn't set permissions  to be world readable.  That seemed to really grab them for some reason.  Unfortunately, the main complaint with that was just lack of time, so we ran through the  whole thing in one and a half hours, and we got as far as just doing basic job submission.  But one thing that was really obvious is with 15 people and a 10 node cluster, is queuing  became really apparent really easily, because one person would submit a five or a 10 node  job and instantly the queue is full.  What I did was actually left up on the projector the output of the watch SQ command, so you  saw every second an update of the queue, and you could see who was in the queue, and they  could go and blame each other and see how long jobs were taking.  A couple of issues we were just running that is I accidentally overwrote subsystem accounts  when I created all the user accounts, and Slurm wouldn't run correctly on the first  day, and that all the Wi-Fi was going through one laptop that was on actually our guest  Wi-Fi because I hadn't registered the laptop properly, and that caused a few internet problems.  Because they were all connecting to the cluster with their laptops for their Wi-Fi, that meant  that all of their internet traffic was also going through that one laptop, and that was  where they were getting the course notes from.  And it meant that they were often flipping backwards and forwards between other networks  or their phones and that main Wi-Fi for the cluster.  I then ran it again for a second time.  This time with a slightly different group, this was approximately 10 people who were  actually doing proper training on how to use our real research cluster.  It was about 10 people with a mix of levels, a mix of staff and students, and this time  I simultaneously ran the real HPC alongside the Raspberry Pi cluster, so people could  choose to use one or the other or both.  It proved to be a really good backup for those who had locked their accounts, of which I  think there were two.  It helped them make it a bit more tangible, but a lot tended to pick one system or the  other and would submit all their commands on one or the other, but I managed to get  the environments close enough that it didn't matter.  They were able to run everything that was in the material on either system.  So in future work, what I would like to do is have a bit better configuration management  for this.  Right now I do have a set of scripts that should theoretically deploy a cluster.  Testing these takes ages because you've got to start from the beginning and build software  and go through the whole deployment process.  I think on the Raspberry Pi ones, compiling the software that I've put in just for that  one module takes about a day.  So it's not a quick process to iterate through the development of this.  The other big difference at the moment between this and any real system is this runs on Raspbian,  which is a Debian derivative.  Pretty much every HPC system I've ever come across runs on some kind of CentOS or Red Hat  Enterprise Linux derivative.  And it would be nice to try and get CentOS to run on there instead of Debian.  As far as I understand, there is an ARM port that will run on the Raspberry Pi, but I haven't  run to trying it yet, and Raspberry Pi is kind of the default Raspberry Pi foundation push.  And I'd also like to focus a bit better on some demos.  The tool I really want to do is we have a large group who analyze satellite imagery.  I'd like to write something around there where someone can type in an address and then see  some satellite imagery of that address and then do some analysis of it to look at how  that land use in that area has changed over time.  The other big area of research for us is in genomics, and we had two PhD students a few  years ago develop something called Monster Lab.  And here you put a set of LEGO bricks together, and the different colors of the LEGO bricks  determine what gene you're looking at.  You run this through a sequencer, which looks at the colors of the bricks, and then determines  what the genetic sequence is and therefore what attributes your monster has.  So for instance, the first LEGO brick is the number of heads, the second one is the number  of legs, the third one is the number of arms, the fourth one is the number of eyes.  And by having different colors, you then get different attributes to your monster.  Right now, all the system does is you read through the brick and it tells you what the  attributes are and the kids then go and draw their monsters.  What I'd really like to do is simulate them and have maybe some kind of physics simulation  that goes and simulates them.  Or to do some gene sequence pattern searching and look for similar monsters that came up  previously, which I suppose is the more realistic scenario to real genomics.  So finally, if you're interested in building your own cluster, I have put up on my GitHub  all of the scripts that I have used to create this.  These are still under development somewhat.  I've basically done one full iteration through them.  Copying the code to the network booting is not fully tested, but you should at least  be able to get a SD card deployed for the head node and for a compute node.  Also up on GitHub, this is actually under my organizational one is the teaching material  that I used.  If you want to know any more about this, then please feel free to email me.  My email address is there on the bottom of the slide.  Do you want to have a question?  James, please thank the speaker.  Before you leave, please keep seated while we do questions.  Any questions for Colin?  Thank you for a very interesting presentation.  I was wondering about support for containers.  I see that coming more and more with singularity.  Could that be added?  I don't know if singularity works on ARM.  I know Docker definitely doesn't, but I'm not sure if singularity can make work on an  ARM system.  Given that we're seeing some real ARM clusters, in the UK there's one called Ismbard at Bristol  University.  That's something that's in the pipework for singularity.  It's definitely a feature that would be nice to add.  Thank you.  Any other questions?  Please stay seated while we have another question.  Is it essential to boot a Raspberry Pi from an image rather than install it in a conventional  way?  Is there any news on an image for CentOS for Raspberry Pi?  You put an image on an SD card.  You have to have an SD card in the older pies to boot from.  My problem is if I want to put 10 images that are identical onto 10 machines, I've got to  get the image right, then copy it.  Then if I make a change, I've got to either redo those changes to every machine or rebuild  the SD card image and then put a new SD card in every one of them.  So by having a network boot, that problem goes away.  Sorry, what was the second bit about CentOS?  Is there any news on a CentOS image for the Raspberry Pi?  I think there is a CentOS image, but I'm not certain.  Right.  Thanks.  There's definitely an ARM image.  In the back, please.  Work together.  I know you can.  Thanks.  Was it a Vessier first choice, or did you also try other file systems in networking  systems like Lustre, GFS, whatever?  I think Lustre might be a bit too much for a Raspberry Pi.  Given the size of the cluster, just NFS is fine.  There's not a performance issue, really.  I'd also need multiple disks if I wanted to use a parallel file system.  Having a single disk being served over a parallel file system won't achieve anything.  I'm not fully aware of HPCs that much, so it might be a newbie question.  But as a user, the PhDs who are using, like they're logging on onto the Raspberry Pi,  do they see any difference instead of just SSH into a just regular node instead of HPC?  Do they experience any different?  Not really.  That's part of the point, is that the experience is very similar to the real system.  The main difference is obviously that it's got less memory, less disk space, and it's  slower.  But that's not immediately obvious when you're just typing commands at an SSH prompt.  So what would be an advantage for them to train on?
rpi_cluster_0_131.png rpi_cluster_132_1127.png rpi_cluster_1128_3119.png rpi_cluster_3120_3803.png rpi_cluster_3804_4175.png rpi_cluster_4176_4787.png rpi_cluster_4788_5231.png rpi_cluster_5232_6203.png rpi_cluster_6204_9803.png rpi_cluster_9804_11183.png rpi_cluster_11184_13931.png rpi_cluster_13932_15227.png rpi_cluster_15228_16343.png rpi_cluster_16344_17507.png rpi_cluster_17508_17555.png rpi_cluster_17556_17951.png rpi_cluster_17952_18467.png rpi_cluster_18468_19463.png rpi_cluster_19464_20987.png rpi_cluster_20988_21983.png rpi_cluster_21984_23051.png rpi_cluster_23052_24515.png rpi_cluster_24516_26387.png rpi_cluster_26388_28691.png rpi_cluster_28692_28787.png rpi_cluster_28788_28847.png rpi_cluster_28848_32171.png rpi_cluster_32172_35243.png rpi_cluster_35244_35386.png