Automated testing of OwnCloud using VirtualBox, Vagrant, Ansible and Apache JMeter
CC Attribution - ShareAlike 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Boer, Jean-Marie de
I will demonstrate how I use VirtualBox, vagrant, ansible and Apache Jmeter to run reproducable performance tests of different versions of OwnCloud. At SURFsara we use this tool to measure performance of a new OwnCloud version before upgrading. In a nutshell, it starts with a level playing field every time, since installation of sources and populating the database are part of the process. We never go have to go back to old results and compare them with new results, that would not give a clear picture. Rather, we can spin up the environment afresh, be it on a laptop or a high-end server, and re-run the test suite for selected versions, giving a fair comparison. I will explain the concepts and tools used, give a live demonstration of a short test run, and will present results of longer test, that actually took several days to complete and consisted of millions of requests.
 All right, so we're a bit early. We're running a bit early, but I guess that's okay. You can all have your beer a little bit quicker. My name is Jean-Marie de Bourg. I work for Surfsara in Amsterdam, where we run a fairly large own cloud installation, nearly 40,000 users, about 270 million files. I worked there for just over two years now, two and a half years. Almost two years ago, we did an upgrade. We were running our own cloud eight at the time. We're now at 10.03. That upgrade at that time, it didn't go quite as planned. We did some functional testing before. Everything seemed to be fine. We did the upgrade. It was also good. But then when we opened the doors, opened the floodgates, and users started coming in, we had some performance issues, actually. We worked on this for a couple of days together with the guys from own cloud. In the end, at that time, we couldn't resolve it, so luckily we could roll back. Later we upgraded to 10, which went fine. Since that time, own cloud has done a lot of improvement on their testing. But at the time, we felt we also needed a way ourselves to have more of an idea of performance impact when upgrading, because when we went from eight to nine, we really had no idea. That's when I started looking in performance testing of different versions of own cloud. As I said, we need a way to assess the performance before we do the upgrade, just for our own piece of mind, basically, and of course, for our users. We also know that the prop find is the most common request that we encounter. We have about 5,000, between 5,000 and 5,000 active users. Most of them are running the sync client. In any minute, that's 10,000 prop finds that we have to handle. When they make some changes, other stuff happens, but the prop find happens all the time. We focused on that specifically. When talking with own cloud, they gave us the indication that the number of shares a user had would have some impact on response time. This is basically the presumptions or the starting point for our testing. Apache JMeter. I don't know if anybody here has ever heard of it before. Nobody? Oh, here it is. Some of the serve side guys, well, I'm not surprised. There's other tools for performance testing. Smashbox is something that I have heard about then and also now it's been mentioned to me. I chose JMeter because I worked with it before. I knew I could get something up and running quite quickly. What is Apache JMeter? It's a project from the Apache Foundation. It's basically an HTTP client that you can program. You can create scenarios and play them. It then measures the response times of the requests that it does for you. You can parse those responses and, for instance, to popular variables that you're going to use later in your test. Use case would be you have a login form. It has a CSRF token that's randomly generated. You can do a request for the form, parse the response, then you have the content of the CSRF token and post that in your next step. It does everything HTTP can do. You can do a prop find. You can do a put, get, delete, anything. You can do loops, conditionals, assertions. You can say I do a request on a certain resource. In my response, I expect a certain string. It's actually quite a powerful tool. Just like Thomas, I'm an adrenaline junkie, so I'm going to give you a live demo. Are we? Okay. Here we are. Yeah. So that looks pretty, well, 1990s. I think it's partly because I'm doing an X session between different versions of Linux. Also I did some scrolling on the screen, but mainly I think when we all went into the new millennium. We had the Apache foundation forgot to bring the Jmeter GUI, so that's still living in those times. So what do we have here? So I have a test plan. I hope you can kind of see this. This is my test plan. It has a thread group. And what I'm going to do is I'm going to start five threads and I'm going to ramp up in five seconds. So basically it's going to do five tests in a five second period. And what is my test? It's actually one HTTP request going to oncloud.com. I'm just going to do a get. So let's run that. So it's running. You can see it's doing my five requests. The status is good. If it would be not a 200, it would give me something that's, it would give a red shield. And I can see here the sample time. So I can see how long each request took. And here at the bottom I can see the average response, the average time it took to get that response. Also latency, how many bytes were sent. Then I can zoom in. I also see my requests here. I can see what is actually the request that I did. I did a get on this URL. And this is what I got back. I got back an HTML page. So this is a great way if you want to do load testing. If you say I want to be able to serve X amount of people in Y amount of time and still guarantee response time that's less than so many milliseconds, you just say I start so many threads in the time that I want to be able to serve those people. So I chose a slightly different approach. Because what I want to do, I don't want to measure the absolute performance of my laptop or any system that I'm testing. I want to test the relative difference between versions of on cloud. So I'm going to run sequential requests. So it's very much the same. But now I have a thread group with just one thread. But inside I have a loop controller. So I'm going to loop five times and then do pretty much the same request. So I can run that. And we'll get very similar results. So this is basically the most simple kind of test you can do in Apache Jmeter. You just do a get or a post and you see how long it took you. OK. So back to this. So as I said, I'm not interested in the absolute performance. That would be interesting if I would be testing our production systems. But because I want to test these upgrades, I want to do the relative performance. What I specifically want to test also is the PHP execution and the database performance. I'm not really interested in the network or the storage. Because when we upgrade, that's all going to stay the same. So that's not going to tell me anything different between those versions. Also I'm not particularly interested in doing any server optimization. I just want to create one baseline and do the different versions against that baseline. And specifically I don't want to do query caching because in this kind of performance I'm doing the same query all the time. Whereas if you have a real workload, you have different clients all the time, different query caching may actually hurt you rather than help you. So there's no query caching. OK. So initially what I did was pretty much like what I just showed you. Prop find in oncloud is actually quite simple. You just do a prop find or what the client is. It was a prop find on a certain webdav endpoint with some authentication. And you could actually do that from the GUI just like I did just now. So we have these servers with acceptance environments where we have different versions of oncloud already running. I would just log in using a browser. Then I would look, OK, what's the value of my cookie? I would put the cookie into JMeter. So basically I just hijacked my browser session. And I would run a test against the different installations. Then I would manipulate the database so I'd have different number of shares. Well, you can already hear it's a lot of work. It's very error prone because I'm manipulating the database. I might make a mistake. I mean, I could see my results were pretty consistent. But it's not really the way you want to do this. And another big disadvantage is I'm testing against some systems. Well, what, a few weeks later, I want to test another nightly build. I have no way of telling if these systems are still comparable. Maybe I've upgraded PHP or the database or I don't know what happened. So I would have to do all those steps again because I couldn't compare what I did now compared to a couple of weeks ago. So it's basically a lot of work, a lot of heavy lifting. And I went into IT because I want to avoid artwork. I want to automate. So automation. So what I want to do is basically not that difficult. It's not rocket science. It's all been done before. It was just a matter of getting the right tools together. So the first obvious step is not to test against our testing systems, but to use a VM, a virtual machine. So there's many hyperfixes you can use. I chose virtual books. You may know it. It's from Oracle. I chose it because it runs on Linux. What I use, it runs on Windows or OS X. You know, it's easy. Just grab it off the shelf and you have it. And it's quite easy to spin up a VM. But it's even easier if you use vagrant. Most of you might know vagrant. It's basically to shell around virtual books where you can just tell it, okay, I want to spin up a VM. This is how much memory you should have. Processors. It can insert an SHH key for you. So you can log in immediately. Set up some port forwarding. Basically just make sure VM usable very quickly. And it has a provisioner. So you can actually even show you what the vagrant file looks like. I have it here. So basically, I have a VM. It has two gigs of RAM. Forward some ports. It has a folder inside the VM, which is actually pointing to a directory on the host. And then at the end, run my Ansible playbook. So Ansible, I use it to bring the VM into a state where I can actually start doing the testing. So basically it's, I need Apache. I need PHP. I need MaraidDB. So I just basically in the playbook it installs that stuff for me. But it does some more things because I want to, the VM is clean. It's just a center with nothing on it. And I want to test on cloud, and I don't want to have to install it every time. So in my playbook, after I set it up, I get my project from GitHub. Then I have to tell, well, what am I going to test? So this is all the stuff I can test right now. Different versions of on cloud. And yes, you can also test next cloud with it. And as of today, I also added 10.0.10 because, well, why not? So this is a disk file. You have to make your own copy. If I did this, then, okay, I'm just testing these particular versions. It also downloads the JMeter binary for me. And it creates a file that I'm going to use for upload testing. And I can specify here how big I want that file to be. Now, in my tests, I use a one byte file because I'm not interested in how long it takes to put it across the network or write it to disk. I just want to see how fast on cloud can process it for me. But Holger said he wanted to do some upload testing. Well, you could put here 100 MB, and it's going to upload 100 MB for you. It would also possible. Yeah, yeah. I haven't tried 100 gigs, but anyway. So when I then do the vagrant provision, it's going to run the playbook. It actually downloads the on cloud sources for me and installs them in the VAR-WWHTML directory. But then it's still not working because when you install on cloud, there's a step you have to take where you have to log in the first time. You have to create your admin user. It initializes the database. So this is actually precooked into my project. So in the project, there's a config file for the different versions, and there's an empty database with just the admin user with password, also admin. But that's all. And the empty database is there so that every time the run starts, it's going to start with that clean database. So now every time I run this thing, I set this up. I have a level playing field for all the versions. And it's also portable because I can run it here, but I can also check out the project on anywhere else where as long as I have virtual box and vagrant, and it's going to run there as well. So then there's one other file I have to edit, which actually tells me how my test, what my test look like. Well, maybe I could go to the next slide because what is my automated test at this time? Well, let me come back to my browser first, actually, because I just started this. I have vagrant running now. So this is an Oncla 10.09. That's actually running in the VM. I will show you if I log in. You can see it's completely fresh. It's the first time I log in. And it's empty. There's just me. And I have 10.0.10 on here as well. It's going to show you the same thing. So what does my test do? The test is created in Jmeter. I will show it to you in a minute. So I need some users. So it's going to create some users. Then I thought, well, if the users are there, I can use the API to also search for them. Let's see. There's another API call. It's going to upload some files for me. It's going to share files. So one user is going to share files with another user. Then it's going to do propfines and it's going to delete the files again. So what does that look like in Jmeter? I will just show you. OK. So I have a property file reader that just reads in any file. I have a bunch of variables, like how many users am I going to create? How many propfines do I want to do? And my test, basically, first I'm going to create those users. If a certain condition is true, so this is actually those specific intervals, it's going to do a search. It's going to loop all those users and upload files for them. It's going to do a bunch of propfines where no shares exist. It's going to start creating the shares and then in the end it loops the users again and deletes the files for them. And you can see, there's authorization manager in Jmeter where you can say, well, I have these users. The user and password is the same, so you actually log in just like you would on, well, actually it's just the same as logging in via a client or a browser. So I'm going to give you a little demo. So I'll take this one here. Let me get my mouse out of the way. Sorry about that. My mouse is slipping. Okay, so when I log into the VM, because it's been provisioned before, there's a little shell script there that has the different versions in it. So it can tell me, well, these are the versions that are available on your VM. If you want to test them, just execute these commands. So I told you about the property file before. This is the property file that's active now. It's going to create three users. Each user is going to do 10 uploads. It's going to create 10 shares after every five. It's going to do some propfines. It only does five propfines. You know, this should be higher numbers for any normal test. You'd create more users. You'd do more requests to filter out the outliers. But this is just as an example. So let's see if it actually works. It's imported to database. I know something is going to happen. Okay, so it's done 50 requests now. Well, it's done the first one already. It's moving on to the next one. 10.09, 10.09. And it goes on to the next one. And this is actually, it's a shell script that's running the same script three times which is passing a different vendor and version. So now it's done. Well, you can see I didn't do this in the GUI this time. So where's all the information that we could see in the GUI before? Well, there's a shared directory that's inside the VM that's actually living on the host. And we can go there. It's right here. So what has happened is the Jmeter test, it has run, it has written its results to a directory on the host. So here I have a CSV file which is basically all that stuff that we saw in the GUI before. I have it now in a CSV file for the different versions. But also you can see here it's written an HTML directory. And here, maybe I'll go to version 10. Version 9 is an old het, right? We go to 10.09. So here you can see all those requests that it just did. It created three users for me in an average of 330 milliseconds. And then it uploaded these files. It did 30. Well, that's correct. You know, we had three users, 10 upload each. And then it did prop finds with different numbers of shares. Now, normally, of course, you use much bigger numbers. You leave it running for a longer amount of time. I think I have here some results. So here I did 81,000 requests. You know, I have 400 users created. I think I did 100 uploads each. Or 200 even. So that's 80,000 uploads. And then it gives me all these numbers. And I actually put them together in a sheet for you. So what do we see? So we still use 8 as a baseline also because my tooling still supports it. And why not? So what do I see? If you look at the, when there is no shares, the difference is not that big. OnCloud 9 is quite a bit slower than 8. I mean, it's still like in the region of 10%. But as the number of shares increases, you can see that the OnCloud 9 actually becomes quite a bit slower. OnCloud 10 is actually a lot better again. But still not quite as fast as OnCloud 8. So there is some work to be done still. This is the slightly larger test. You can see the same pattern. But actually here you see that 10.009 is actually a bit quicker again than 10.003. So we're moving in the right direction. I have not had time yet to test this against 10.010. But that would be interesting. What I would like to do, what I would have liked to have presented to you today is a test where we have, let's say, 4,000 users and a million files. Unfortunately, I didn't manage to do it with the current tooling. JMeter is really good for load testing. So you run a load test for a relatively short amount of time. I have tried creating a whole bunch of users and having a test run for several days. But I wasn't quite happy with the results. There were two fluctuating. I must say that what you see here, this pattern where you have OnCloud 8.00, then OnCloud 9.00 becomes quite a bit slower and OnCloud 10.00 is a lot better. I have seen that in pretty much all my tests. So the next thing I'm working on right now, some initial conclusions. Yeah, we can definitely see the shares influence propfined timings. 10.00 is better than 9.00, still OnCloud 8.00 is from a purely performance point of view on this test. It's still the fastest. But there's still room to expand my testing suite. Because of course, when you are testing, you only see what you are looking for. So there may be a lot of other stuff that you're not looking for, which is good or bad. You just don't know. I need a better method to test really large data sets. Now there is something in my project, which I invite you all to look at it. And if you have suggestions, please. What it does right now is it reads the empty database when you start the test. But then it looks for an SQL file with a certain name. It's the same. It's just the vendor version dash extra dot SQL. Then it's going to load that in and then run the tests again with this extra SQL file applied. So actually what I just have to do is create some reference databases that are good for making this fair comparison. So I have to do more analysis on what does a big database look like and how can I, we have one, but it shouldn't be that big. I know, but ours is a bit, because that is going to take a long time to load it in. So that's something I'm working on at the moment and also expanding the tests. You are looking for different things that we could be testing. So this is also the URL where you can download the project. It's really that simple. You just check it out. You change those two files and you can run it. And it's going to run, well, I guarantee you it's going to run pretty much flawlessly. And that really was my talk. So I'm also quite quick. Any questions? Yeah, well, you have to, I started this two years ago and at the time there was some Docker infrastructure available. But again, this for me was just a way to get on the ground quickly, to get something up and running quickly. But definitely I think the underlying virtualization technology or whether it's virtual machine or containerization. We'll take up the machine. It might be more, like, to be a test-regal thing. Well, good point. Good point. Well, we are at the other side of moving towards Docker. We have some Docker stuff running at the moment. We have on-cloud running in Docker. And Tom and Vibranth are going to tell you all about it on Thursday. So that's definitely something that's interesting to see how I can take this into the future and maybe towards a containerized approach. I think the differences there are not that big. I think so, if you look at, even if you have 10,000 users, that's still a relatively small table. I think it's more like file cache in combination with shares. That's where you're going to find more of a performance. So you can see that even if, as the number of users go up, not a lot happens to the response time of that request. So this is for the sharing API. So this is what you do when you start typing a username in the web interface. I'm going to be here for the rest of the week. Another question is. What are the imperfections for an adminwhy always, usually the one doesn't meet, are I'm not sure. Well, I mean, there is the, like the percentiles, if that's what you mean. Yeah, errors is, in this case, it's zero because they all give a 200 response code. That's more if you set up your testing if you create some mistakes in your testing scenario. That's true, but I mean, like if you look here, you see that there is, if you look here, for instance, there's 182 millisecond response, but that's in a 99%. So that's just going to be one. But this is why you have to have a reasonable number of samples. So I do 50 samples. So, you know, definitely some of them may be slow. You know, you can do more to filter out the outliers, but if you would just do two samples, you wouldn't really have a reliable result. I think 50 is a good number, you know. So the whole question is how do we test more? No. Well, thank you. Because how do you make it for the second? So like, how do you, like, you have this agent that they are, like, some sort of excuse, but is it like that? So you always run it with the same? Well, not necessarily, because I could run this on a machine with, let's say, a faster processor, and all the times would go down. But relatively, between the versions, you would have the same relative performance. You know, so that's why I'm not interested in the absolute numbers, but I'm just interested in the differences between the versions. So you do see, like, you know, like, on zero shares, 9.1 is going to be about, you know, 7, 8, 9% slower than 8. So if it's 100, it's going to be 110. If it's 200 milliseconds, because it's a slow processor, it's going to be 220 on the other version. So you're going to just, like, it's like, it could be a different kind of, you know, like, for a source that you want to take the time to run. I don't think so, because what I'm doing is I'm a single client, you know, so I'm, it's only one Apache process basically running at the same time. So it's not going to run out of memory, you know, it would be a bit quicker if I would, I think, yeah, I think not. I think in this current, because it's a very small scale test from a client point of view. So you don't need a lot of resources. Yeah, it's difficult. But at the same time, you will find people as a raw web server or run all the time. Yeah, that's, of course, there's always going to be some, some impact, but then again, there's going to be the same again for each test run, because you don't do the same thing. I think what could also happen is that the, the hosts that you run the VM on is that can also impact, of course, the VM, you know, but you know, I'm not, I'm not trying to find the Higgs boson here. I'm just getting a pretty good, pretty good idea. And, you know, from, and like I say, these relative performance, I whenever I run the test is pretty much what I see. So yeah, don't tie me down on one or two percent, you know, but between five, five percent, you know, that's, yeah, that's my maximum margin of error, I would say. Just as easy as setting up the installer and that's really good for the whole, you know, if you like, I can help you set it up later, you know, and you can, you can play around. Yes. Well, like I said, I worked with Jmeter before and I knew it can do this stuff and I knew I would be able to just get something up quickly. And at the time I thought, okay, Smashbox seems interesting, but it probably has a learning curve. That's also what I heard. So, you know, I, I'm pragmatic, you know, I just, well, this is something I know, I know it's going to give me some useful results. That's why I chose it. So, it's interesting actually to correlate the two. Let's see, you know, what does this test suite show and then correlate it with, with what Smashbox shows. And ideally it should give us pretty much the same picture. And if it doesn't, then that's interesting. Yeah, like I said, I mean, it started just, I just wanted to do prop finds, you know, that's all I wanted to do. But then I thought, well, if I'm going to do it in a fair way, I need a level playing field. And that's when I got to the stage where I said, well, I have an empty database. I create the users, I do the uploads. Just so that I know every time I test this is the same. You know, I don't know what that's like with the Smashbox testing. If that also like starts off with the same base situation every time or whether you run against a production system or. Definitely I'd be interested in, you know, discussing and comparing results. And you probably have some tests that would be really interesting to integrate into this as well. Well, the thing is you have to authenticate, you know, so I guess I could do it. I have done this kind of thing before where I would have a list of user name passwords that I would use to authenticate. But here I can do it because I know what user accounts exist in the system because I just created them, you know, in a production system. Yeah, for us, it would be a little bit difficult because we use a shibbolet, you know, so. And we don't have functional accounts, so we only have like real accounts. So I could only use like real accounts. I have actually just for a hobby project. I have created a Jmeter test that does a shibbolet login. So it goes to the to the serve drive homepage. Then you a token is generated. You get redirected to the IDP with that token. There you get a form which has more tokens in it. You post a form with your token that you got from from from serve drive. You get redirected back, you know, with yet another token. In the end you log in and Jamie can actually do that because you can at every step, you know, like say I want to parse these tokens and then, you know, bring them with me to the next. The next step in the login flow. It could be done. But like I say, the main problem is we don't have the accounts to test on on our production systems. Plus, you would have to do it at the right time. I have done it with other systems, systems for schools that are also a lot of users, but we would do this at night because we knew the children aren't at school at night for the system isn't being used. So if you're actually load testing your your system, you're going to impact your real users, possibly, because at some point you're going to run into the into that, you know, curve where you start degrading the performance of your system because you start overloading it. I suppose so. I said, yeah, yeah, yeah, because right now it's in the in the YAML file. It just tells you where you should where it should download the sources. So I have it here. So it just tells me, you know, this, this just says where it's going to get that tar file from that is going to install. You could change this with anything, even a local host and, you know, download another source package. You could also just go into the VM and, and, you know, change your code there as long as you don't reprovision, it's not going to, it's not going to, you know, or as long as you don't reprovision or destroy the VM. You could work there. I mean, it's a bit volatile, but you could do that. Well, right now it expects that the tar dot bzip file. So you would have to be in that format. But yeah, I guess you'd have to package that yourself. Well, like I said, the server part doesn't really interest me that much. Like I said, what I install is really simple. I just do ph php seven Apache without any performance tuning or tweaks. I could discuss what things I could do on the server side to get more useful results. Yeah, no, that's, that's actually, that's a good point. That's something I definitely want to look at is enabling apps or making it possible to say I want to install these apps and with this configuration. Yeah. I think the challenge there is to keep it level because this is just a plain community version without any bells and whistles. It's the comparison is fairly fair. Maybe when you start having different apps for different versions, maybe you also run into some compatibility issues, because now the test that runs against all these like eight or nine versions, because they're all like compatible on that on that level. So. Yeah, yep. Yeah, that's the only that's the only performing thing. Thank you.
 All right, so we're a bit early.  We're running a bit early, but I guess that's okay.  You can all have your beer a little bit quicker.  My name is Jean-Marie de Bourg.  I work for Surfsara in Amsterdam, where we run a fairly large own cloud installation,  nearly 40,000 users, about 270 million files.  I worked there for just over two years now, two and a half years.  Almost two years ago, we did an upgrade.  We were running our own cloud eight at the time.  We're now at 10.03.  That upgrade at that time, it didn't go quite as planned.  We did some functional testing before.  Everything seemed to be fine.  We did the upgrade.  It was also good.  But then when we opened the doors, opened the floodgates, and users started coming in,  we had some performance issues, actually.  We worked on this for a couple of days together with the guys from own cloud.  In the end, at that time, we couldn't resolve it, so luckily we could roll back.  Later we upgraded to 10, which went fine.  Since that time, own cloud has done a lot of improvement on their testing.  But at the time, we felt we also needed a way ourselves to have more of an idea of performance  impact when upgrading, because when we went from eight to nine, we really had no idea.  That's when I started looking in performance testing of different versions of own cloud.  As I said, we need a way to assess the performance before we do the upgrade, just for our own  piece of mind, basically, and of course, for our users.  We also know that the prop find is the most common request that we encounter.  We have about 5,000, between 5,000 and 5,000 active users.  Most of them are running the sync client.  In any minute, that's 10,000 prop finds that we have to handle.  When they make some changes, other stuff happens, but the prop find happens all the time.  We focused on that specifically.  When talking with own cloud, they gave us the indication that the number of shares a  user had would have some impact on response time.  This is basically the presumptions or the starting point for our testing.  Apache JMeter.  I don't know if anybody here has ever heard of it before.  Nobody?  Oh, here it is.  Some of the serve side guys, well, I'm not surprised.  There's other tools for performance testing.  Smashbox is something that I have heard about then and also now it's been mentioned to me.  I chose JMeter because I worked with it before.  I knew I could get something up and running quite quickly.  What is Apache JMeter?  It's a project from the Apache Foundation.  It's basically an HTTP client that you can program.  You can create scenarios and play them.  It then measures the response times of the requests that it does for you.  You can parse those responses and, for instance, to popular variables that you're going to  use later in your test.  Use case would be you have a login form.  It has a CSRF token that's randomly generated.  You can do a request for the form, parse the response, then you have the content of the  CSRF token and post that in your next step.  It does everything HTTP can do.  You can do a prop find.  You can do a put, get, delete, anything.  You can do loops, conditionals, assertions.  You can say I do a request on a certain resource.  In my response, I expect a certain string.  It's actually quite a powerful tool.  Just like Thomas, I'm an adrenaline junkie, so I'm going to give you a live demo.  Are we?  Okay.  Here we are.  Yeah.  So that looks pretty, well, 1990s.  I think it's partly because I'm doing an X session between different versions of Linux.  Also I did some scrolling on the screen, but mainly I think when we all went into the  new millennium.  We had the Apache foundation forgot to bring the Jmeter GUI, so that's still living in  those times.  So what do we have here?  So I have a test plan.  I hope you can kind of see this.  This is my test plan.  It has a thread group.  And what I'm going to do is I'm going to start five threads and I'm going to ramp up in five  seconds.  So basically it's going to do five tests in a five second period.  And what is my test?  It's actually one HTTP request going to oncloud.com.  I'm just going to do a get.  So let's run that.  So it's running.  You can see it's doing my five requests.  The status is good.  If it would be not a 200, it would give me something that's, it would give a red shield.  And I can see here the sample time.  So I can see how long each request took.  And here at the bottom I can see the average response, the average time it took to get  that response.  Also latency, how many bytes were sent.  Then I can zoom in.  I also see my requests here.  I can see what is actually the request that I did.  I did a get on this URL.  And this is what I got back.  I got back an HTML page.  So this is a great way if you want to do load testing.  If you say I want to be able to serve X amount of people in Y amount of time and still guarantee  response time that's less than so many milliseconds, you just say I start so many threads in the  time that I want to be able to serve those people.  So I chose a slightly different approach.  Because what I want to do, I don't want to measure the absolute performance of my laptop  or any system that I'm testing.  I want to test the relative difference between versions of on cloud.  So I'm going to run sequential requests.  So it's very much the same.  But now I have a thread group with just one thread.  But inside I have a loop controller.  So I'm going to loop five times and then do pretty much the same request.  So I can run that.  And we'll get very similar results.  So this is basically the most simple kind of test you can do in Apache Jmeter.  You just do a get or a post and you see how long it took you.  OK.  So back to this.  So as I said, I'm not interested in the absolute performance.  That would be interesting if I would be testing our production systems.  But because I want to test these upgrades, I want to do the relative performance.  What I specifically want to test also is the PHP execution and the database performance.  I'm not really interested in the network or the storage.  Because when we upgrade, that's all going to stay the same.  So that's not going to tell me anything different between those versions.  Also I'm not particularly interested in doing any server optimization.  I just want to create one baseline and do the different versions against that baseline.  And specifically I don't want to do query caching because in this kind of performance  I'm doing the same query all the time.  Whereas if you have a real workload, you have different clients all the time, different  query caching may actually hurt you rather than help you.  So there's no query caching.  OK.  So initially what I did was pretty much like what I just showed you.  Prop find in oncloud is actually quite simple.  You just do a prop find or what the client is.  It was a prop find on a certain webdav endpoint with some authentication.  And you could actually do that from the GUI just like I did just now.  So we have these servers with acceptance environments where we have different versions of oncloud  already running.  I would just log in using a browser.  Then I would look, OK, what's the value of my cookie?  I would put the cookie into JMeter.  So basically I just hijacked my browser session.  And I would run a test against the different installations.  Then I would manipulate the database so I'd have different number of shares.  Well, you can already hear it's a lot of work.  It's very error prone because I'm manipulating the database.  I might make a mistake.  I mean, I could see my results were pretty consistent.  But it's not really the way you want to do this.  And another big disadvantage is I'm testing against some systems.  Well, what, a few weeks later, I want to test another nightly build.  I have no way of telling if these systems are still comparable.  Maybe I've upgraded PHP or the database or I don't know what happened.  So I would have to do all those steps again because I couldn't compare what I did now  compared to a couple of weeks ago.  So it's basically a lot of work, a lot of heavy lifting.  And I went into IT because I want to avoid artwork.  I want to automate.  So automation.  So what I want to do is basically not that difficult.  It's not rocket science.  It's all been done before.  It was just a matter of getting the right tools together.  So the first obvious step is not to test against our testing systems, but to use a VM, a virtual  machine.  So there's many hyperfixes you can use.  I chose virtual books.  You may know it.  It's from Oracle.  I chose it because it runs on Linux.  What I use, it runs on Windows or OS X.  You know, it's easy.  Just grab it off the shelf and you have it.  And it's quite easy to spin up a VM.  But it's even easier if you use vagrant.  Most of you might know vagrant.  It's basically to shell around virtual books where you can just tell it, okay, I want to  spin up a VM.  This is how much memory you should have.  Processors.  It can insert an SHH key for you.  So you can log in immediately.  Set up some port forwarding.  Basically just make sure VM usable very quickly.  And it has a provisioner.  So you can actually even show you what the vagrant file looks like.  I have it here.  So basically, I have a VM.  It has two gigs of RAM.  Forward some ports.  It has a folder inside the VM, which is actually pointing to a directory on the host.  And then at the end, run my Ansible playbook.  So Ansible, I use it to bring the VM into a state where I can actually start doing the  testing.  So basically it's, I need Apache.  I need PHP.  I need MaraidDB.  So I just basically in the playbook it installs that stuff for me.  But it does some more things because I want to, the VM is clean.  It's just a center with nothing on it.  And I want to test on cloud, and I don't want to have to install it every time.  So in my playbook, after I set it up, I get my project from GitHub.  Then I have to tell, well, what am I going to test?  So this is all the stuff I can test right now.  Different versions of on cloud.  And yes, you can also test next cloud with it.  And as of today, I also added 10.0.10 because, well, why not?  So this is a disk file.  You have to make your own copy.  If I did this, then, okay, I'm just testing these particular versions.  It also downloads the JMeter binary for me.  And it creates a file that I'm going to use for upload testing.  And I can specify here how big I want that file to be.  Now, in my tests, I use a one byte file because I'm not interested in how long it takes to  put it across the network or write it to disk.  I just want to see how fast on cloud can process it for me.  But Holger said he wanted to do some upload testing.  Well, you could put here 100 MB, and it's going to upload 100 MB for you.  It would also possible.  Yeah, yeah.  I haven't tried 100 gigs, but anyway.  So when I then do the vagrant provision, it's going to run the playbook.  It actually downloads the on cloud sources for me and installs them in the VAR-WWHTML  directory.  But then it's still not working because when you install on cloud, there's a step you have  to take where you have to log in the first time.  You have to create your admin user.  It initializes the database.  So this is actually precooked into my project.  So in the project, there's a config file for the different versions, and there's an empty  database with just the admin user with password, also admin.  But that's all.  And the empty database is there so that every time the run starts, it's going to start with  that clean database.  So now every time I run this thing, I set this up.  I have a level playing field for all the versions.  And it's also portable because I can run it here, but I can also check out the project  on anywhere else where as long as I have virtual box and vagrant, and it's going to run there  as well.  So then there's one other file I have to edit, which actually tells me how my test, what  my test look like.  Well, maybe I could go to the next slide because what is my automated test at this time?  Well, let me come back to my browser first, actually, because I just started this.  I have vagrant running now.  So this is an Oncla 10.09.  That's actually running in the VM.  I will show you if I log in.  You can see it's completely fresh.  It's the first time I log in.  And it's empty.  There's just me.  And I have 10.0.10 on here as well.  It's going to show you the same thing.  So what does my test do?  The test is created in Jmeter.  I will show it to you in a minute.  So I need some users.  So it's going to create some users.  Then I thought, well, if the users are there, I can use the API to also search for them.  Let's see.  There's another API call.  It's going to upload some files for me.  It's going to share files.  So one user is going to share files with another user.  Then it's going to do propfines and it's going to delete the files again.  So what does that look like in Jmeter?  I will just show you.  OK.  So I have a property file reader that just reads in any file.  I have a bunch of variables, like how many users am I going to create?  How many propfines do I want to do?  And my test, basically, first I'm going to create those users.  If a certain condition is true, so this is actually those specific intervals, it's going  to do a search.  It's going to loop all those users and upload files for them.  It's going to do a bunch of propfines where no shares exist.  It's going to start creating the shares and then in the end it loops the users again and  deletes the files for them.  And you can see, there's authorization manager in Jmeter where you can say, well, I have  these users.  The user and password is the same, so you actually log in just like you would on, well,  actually it's just the same as logging in via a client or a browser.  So I'm going to give you a little demo.  So I'll take this one here.  Let me get my mouse out of the way.  Sorry about that.  My mouse is slipping.  Okay, so when I log into the VM, because it's been provisioned before, there's a little  shell script there that has the different versions in it.  So it can tell me, well, these are the versions that are available on your VM.  If you want to test them, just execute these commands.  So I told you about the property file before.  This is the property file that's active now.  It's going to create three users.  Each user is going to do 10 uploads.  It's going to create 10 shares after every five.  It's going to do some propfines.  It only does five propfines.  You know, this should be higher numbers for any normal test.  You'd create more users.  You'd do more requests to filter out the outliers.  But this is just as an example.  So let's see if it actually works.  It's imported to database.  I know something is going to happen.  Okay, so it's done 50 requests now.  Well, it's done the first one already.  It's moving on to the next one.  10.09, 10.09.  And it goes on to the next one.  And this is actually, it's a shell script that's running the same script three times  which is passing a different vendor and version.  So now it's done.  Well, you can see I didn't do this in the GUI this time.  So where's all the information that we could see in the GUI before?  Well, there's a shared directory that's inside the VM that's actually living on the host.  And we can go there.  It's right here.  So what has happened is the Jmeter test, it has run, it has written its results to a directory  on the host.  So here I have a CSV file which is basically all that stuff that we saw in the GUI before.  I have it now in a CSV file for the different versions.  But also you can see here it's written an HTML directory.  And here, maybe I'll go to version 10.  Version 9 is an old het, right?  We go to 10.09.  So here you can see all those requests that it just did.  It created three users for me in an average of 330 milliseconds.  And then it uploaded these files.  It did 30.  Well, that's correct.  You know, we had three users, 10 upload each.  And then it did prop finds with different numbers of shares.  Now, normally, of course, you use much bigger numbers.  You leave it running for a longer amount of time.  I think I have here some results.  So here I did 81,000 requests.  You know, I have 400 users created.  I think I did 100 uploads each.  Or 200 even.  So that's 80,000 uploads.  And then it gives me all these numbers.  And I actually put them together in a sheet for you.  So what do we see?  So we still use 8 as a baseline also because my tooling still supports it.  And why not?  So what do I see?  If you look at the, when there is no shares, the difference is not that big.  OnCloud 9 is quite a bit slower than 8.  I mean, it's still like in the region of 10%.  But as the number of shares increases, you can see that the OnCloud 9 actually becomes  quite a bit slower.  OnCloud 10 is actually a lot better again.  But still not quite as fast as OnCloud 8.  So there is some work to be done still.  This is the slightly larger test.  You can see the same pattern.  But actually here you see that 10.009 is actually a bit quicker again than 10.003.  So we're moving in the right direction.  I have not had time yet to test this against 10.010.  But that would be interesting.  What I would like to do, what I would have liked to have presented to you today is a  test where we have, let's say, 4,000 users and a million files.  Unfortunately, I didn't manage to do it with the current tooling.  JMeter is really good for load testing.  So you run a load test for a relatively short amount of time.  I have tried creating a whole bunch of users and having a test run for several days.  But I wasn't quite happy with the results.  There were two fluctuating.  I must say that what you see here, this pattern where you have OnCloud 8.00, then OnCloud  9.00 becomes quite a bit slower and OnCloud 10.00 is a lot better.  I have seen that in pretty much all my tests.  So the next thing I'm working on right now, some initial conclusions.  Yeah, we can definitely see the shares influence propfined timings.  10.00 is better than 9.00, still OnCloud 8.00 is from a purely performance point of view  on this test.  It's still the fastest.  But there's still room to expand my testing suite.  Because of course, when you are testing, you only see what you are looking for.  So there may be a lot of other stuff that you're not looking for, which is good or bad.  You just don't know.  I need a better method to test really large data sets.  Now there is something in my project, which I invite you all to look at it.  And if you have suggestions, please.  What it does right now is it reads the empty database when you start the test.  But then it looks for an SQL file with a certain name.  It's the same.  It's just the vendor version dash extra dot SQL.  Then it's going to load that in and then run the tests again with this extra SQL file  applied.  So actually what I just have to do is create some reference databases that are good for  making this fair comparison.  So I have to do more analysis on what does a big database look like and how can I, we  have one, but it shouldn't be that big.  I know, but ours is a bit, because that is going to take a long time to load it in.  So that's something I'm working on at the moment and also expanding the tests.  You are looking for different things that we could be testing.  So this is also the URL where you can download the project.  It's really that simple.  You just check it out.  You change those two files and you can run it.  And it's going to run, well, I guarantee you it's going to run pretty much flawlessly.  And that really was my talk.  So I'm also quite quick.  Any questions?  Yeah, well, you have to, I started this two years ago and at the time there was some Docker  infrastructure available.  But again, this for me was just a way to get on the ground quickly, to get something up  and running quickly.  But definitely I think the underlying virtualization technology or whether it's virtual machine  or containerization.  We'll take up the machine.  It might be more, like, to be a test-regal thing.  Well, good point.  Good point.  Well, we are at the other side of moving towards Docker.  We have some Docker stuff running at the moment.  We have on-cloud running in Docker.  And Tom and Vibranth are going to tell you all about it on Thursday.  So that's definitely something that's interesting to see how I can take this into the future  and maybe towards a containerized approach.  I think the differences there are not that big.  I think so, if you look at, even if you have 10,000 users, that's still a relatively small  table.  I think it's more like file cache in combination with shares.  That's where you're going to find more of a performance.  So you can see that even if, as the number of users go up, not a lot happens to the response  time of that request.  So this is for the sharing API.  So this is what you do when you start typing a username in the web interface.  I'm going to be here for the rest of the week.  Another question is.  What are the imperfections for an adminwhy always, usually the one doesn't meet, are  I'm not sure.  Well, I mean, there is the, like the percentiles, if that's what you mean.  Yeah, errors is, in this case, it's zero because they all give a 200 response code.  That's more if you set up your testing if you create some mistakes in your testing scenario.  That's true, but I mean, like if you look here, you see that there is, if you look here, for instance, there's 182 millisecond response,  but that's in a 99%.  So that's just going to be one.  But this is why you have to have a reasonable number of samples.  So I do 50 samples.  So, you know, definitely some of them may be slow.  You know, you can do more to filter out the outliers, but if you would just do two samples, you wouldn't really have a reliable result.  I think 50 is a good number, you know.  So the whole question is how do we test more?  No.  Well, thank you.  Because how do you make it for the second?  So like, how do you, like, you have this agent that they are, like, some sort of excuse, but is it like that?  So you always run it with the same?  Well, not necessarily, because I could run this on a machine with, let's say, a faster processor, and all the times would go down.  But relatively, between the versions, you would have the same relative performance.  You know, so that's why I'm not interested in the absolute numbers, but I'm just interested in the differences between the versions.  So you do see, like, you know, like, on zero shares, 9.1 is going to be about, you know, 7, 8, 9% slower than 8.  So if it's 100, it's going to be 110.  If it's 200 milliseconds, because it's a slow processor, it's going to be 220 on the other version.  So you're going to just, like, it's like, it could be a different kind of, you know, like, for a source that you want to take the time to run.  I don't think so, because what I'm doing is I'm a single client, you know, so I'm, it's only one Apache process basically running at the same time.  So it's not going to run out of memory, you know, it would be a bit quicker if I would, I think, yeah, I think not.  I think in this current, because it's a very small scale test from a client point of view.  So you don't need a lot of resources.  Yeah, it's difficult.  But at the same time, you will find people as a raw web server or run all the time.  Yeah, that's, of course, there's always going to be some, some impact, but then again, there's going to be the same again for each test run,  because you don't do the same thing.  I think what could also happen is that the, the hosts that you run the VM on is that can also impact, of course, the VM, you know,  but you know, I'm not, I'm not trying to find the Higgs boson here.  I'm just getting a pretty good, pretty good idea.  And, you know, from, and like I say, these relative performance, I whenever I run the test is pretty much what I see.  So yeah, don't tie me down on one or two percent, you know, but between five, five percent, you know, that's, yeah, that's my maximum margin of error, I would say.  Just as easy as setting up the installer and that's really good for the whole, you know,  if you like, I can help you set it up later, you know, and you can, you can play around.  Yes.  Well, like I said, I worked with Jmeter before and I knew it can do this stuff and I knew I would be able to just get something up quickly.  And at the time I thought, okay, Smashbox seems interesting, but it probably has a learning curve.  That's also what I heard.  So, you know, I, I'm pragmatic, you know, I just, well, this is something I know, I know it's going to give me some useful results.  That's why I chose it.  So, it's interesting actually to correlate the two. Let's see, you know, what does this test suite show and then correlate it with, with what Smashbox shows.  And ideally it should give us pretty much the same picture.  And if it doesn't, then that's interesting.  Yeah, like I said, I mean, it started just, I just wanted to do prop finds, you know, that's all I wanted to do.  But then I thought, well, if I'm going to do it in a fair way, I need a level playing field.  And that's when I got to the stage where I said, well, I have an empty database.  I create the users, I do the uploads.  Just so that I know every time I test this is the same.  You know, I don't know what that's like with the Smashbox testing.  If that also like starts off with the same base situation every time or whether you run against a production system or.  Definitely I'd be interested in, you know, discussing and comparing results.  And you probably have some tests that would be really interesting to integrate into this as well.  Well, the thing is you have to authenticate, you know, so I guess I could do it. I have done this kind of thing before where I would have a list of user name passwords that I would use to authenticate.  But here I can do it because I know what user accounts exist in the system because I just created them, you know, in a production system.  Yeah, for us, it would be a little bit difficult because we use a shibbolet, you know, so.  And we don't have functional accounts, so we only have like real accounts.  So I could only use like real accounts.  I have actually just for a hobby project.  I have created a Jmeter test that does a shibbolet login.  So it goes to the to the serve drive homepage.  Then you a token is generated.  You get redirected to the IDP with that token.  There you get a form which has more tokens in it.  You post a form with your token that you got from from from serve drive.  You get redirected back, you know, with yet another token.  In the end you log in and Jamie can actually do that because you can at every step, you know, like say I want to parse these tokens and then, you know, bring them with me to the next.  The next step in the login flow.  It could be done. But like I say, the main problem is we don't have the accounts to test on on our production systems.  Plus, you would have to do it at the right time.  I have done it with other systems, systems for schools that are also a lot of users, but we would do this at night because we knew the children aren't at school at night for the system isn't being used.  So if you're actually load testing your your system, you're going to impact your real users, possibly, because at some point you're going to run into the into that, you know, curve where you start degrading the performance of your system because you start overloading it.  I suppose so.  I said, yeah, yeah, yeah, because right now it's in the in the YAML file. It just tells you where you should where it should download the sources.  So I have it here.  So it just tells me, you know, this, this just says where it's going to get that tar file from that is going to install.  You could change this with anything, even a local host and, you know, download another source package.  You could also just go into the VM and, and, you know, change your code there as long as you don't reprovision, it's not going to, it's not going to, you know, or as long as you don't reprovision or destroy the VM.  You could work there. I mean, it's a bit volatile, but you could do that.  Well, right now it expects that the tar dot bzip file. So you would have to be in that format.  But yeah, I guess you'd have to package that yourself.  Well, like I said, the server part doesn't really interest me that much.  Like I said, what I install is really simple. I just do ph php seven Apache without any performance tuning or tweaks.  I could discuss what things I could do on the server side to get more useful results.  Yeah, no, that's, that's actually, that's a good point. That's something I definitely want to look at is enabling apps or making it possible to say I want to install these apps and with this configuration.  Yeah.  I think the challenge there is to keep it level because this is just a plain community version without any bells and whistles.  It's the comparison is fairly fair. Maybe when you start having different apps for different versions, maybe you also run into some compatibility issues,  because now the test that runs against all these like eight or nine versions, because they're all like compatible on that on that level.  So.  Yeah, yep.  Yeah, that's the only that's the only performing thing.  Thank you.
ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_0_71.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_72_203.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_204_383.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_384_827.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_828_1043.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_1044_1391.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_1392_2303.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_2304_2555.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_2556_8255.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_8256_10907.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_10908_11171.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_11172_17747.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_17748_18779.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_18780_22979.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_22980_24467.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_24468_24719.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_24720_26267.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_26268_27371.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_27372_27443.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_27444_28163.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_28164_28547.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_28548_29531.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_29532_31103.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31104_31259.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31260_31307.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31308_31355.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31356_31439.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31440_31547.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31548_31619.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31620_31667.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31668_31787.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31788_31979.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_31980_32123.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_32124_32567.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_32568_32711.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_32712_32759.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_32760_32843.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_32844_33335.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_33336_39779.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_39780_40451.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_40452_42155.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_42156_42707.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_42708_42755.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_42756_42827.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_42828_44531.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_44532_50231.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_50232_51023.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_51024_54419.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_54420_54767.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_54768_55079.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_55080_55139.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_55140_56171.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_56172_56375.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_56376_56435.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_56436_57383.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_57384_58139.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_58140_58691.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_58692_61235.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_61236_61763.png ownCloud_conference_2018_Performance_testing_of_OwnCloud_using_VirtualBox_and_Apache_JMeter__a50rSE0_61764_63679.png