Spoilers and Counter-Spoilers
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bernardini, Riccardo
Organizing quality software competitions In this talk we describe an approach for organizing programming competitions that aim to reward software qualities such as readability, maintainability, and so on. At my university we are considering the idea of organizing a software competition with the objective of promoting, beside programming ingenuity, the production of software of good quality, that is, software that it is easy to read, maintain, modify, debug, ..., Such qualities are, by their own nature, quite "fuzzy" and ill-defined, so measuring them in an objective and quantitative way is not easy. In order to solve this problem we figured out an approach based on a competition between "spoilers" that introduce bugs in the software and "counter-spoilers" that try to debug them. Briefly, each spoiler is given a set of programs and s/he must modify them by introducing bugs, as subtle and difficult to find as s/he can. Each counter-spoiler receives a set of programs and s/he must find the bug(s) in them as fast as s/he can. Finally, every program is given to many different spoilers to get many different spoiled versions. The competition turns out to be, actually, three competitions in one since we reward The best spoiler, that is, the one that introduces the most difficult to find bugs. The best counter-spoiler, that is, the one that is fastest in finding the bugs. The best program, that is, the one that is most "resilient" to bug insertion. The "untangling" of the times required for debugging into a measure of spoiler, counter-spoiler and program quality is done by using a statistical approach (that we plan to describe in detail in the talk). The aim of the talk is to describe in detail the technique, stimulate feedback from the community and, also, search for volunteers spoilers and counter-spoilers.
 Thank you. Good afternoon to everyone. I am Ricardo Bernardini. I work at the University of Udine and I guess you can smell my dressing code and my slides will smell like academic, but never mind. I'm here to talk a little bit about an idea that we got at the University of Udine to organize a competition that rewards software maintainability and easy of reading and stuff like that. So, a brief outline. First, what is my objective with this talk. Then, what is the problem? What is the solution? What we are doing right now? And some conclusion. This is very academic. So, my objective is to tell you about this idea about how to organize competition that rewards software quality. And why do I want to do that? Because first of all, I guess it's something interesting to share. Also, we are still working on this stuff, so we are open to feedback. We will be very happy to hear feedback from you. And also, of course, we are also searching for competitors because a competition without competitors is not going anywhere. So, the problem. Our idea was, you know, usually people put a strong emphasis on quality, software quality is like efficiency or maybe how fast you can code. There is this well-known, actively extreme programming competition. Or maybe also ingenuity of solution or stuff like that. But there are not many competition that rewards other types of software that, in our opinion, are also very important. This is how good a software is written. How easy is to maintain, to read, to modify? I have some experience in modifying or pursuing code, both to correct maybe minor bugs. I have some times to modify them for my work. I work in digital signal processing. And I have experience with, for example, a very well-known program that I wanted to modify for using my research. And the code is so confusing that in the end I gave up. So, we decided to try to create a competition that rewards also this kind of quality. Those kind of competition are not common. And there is a very good way, reason. It's easy to measure how fast you can code. I say code a shell sort algorithm. Start, stop, 20 minutes. Nice. That's easy. It's easy to measure how efficient the program is. Start, stop. That's nice. But how can you measure something like how easy is to read the program? How easy is to change to debug it? The first solution that comes to mind is peer review. Having someone look at the code and say, oh, this is a very well-written, this is not so well-written, so. But, unfortunately, peer review is quite subjective. And people have usually very strong opinion about coding styles. For example, Hungarian notation. Just let me do a check. How many of you do know what Hungarian notation is? OK. Very good. How many of you do love it? Do you like it? A few hands. OK. I guess the other one just hated it. OK. I'm going to the second group, by the way. Casing convention. Do you prefer underscores? Cable convention, mixed convention. And also there is the so-called section title convention that makes uppercase, nonce, and verbs, lowercase, everything else. Code blocks. How do you write blocks? This way, this way, with a large indent, small indent, using tab, using spaces. If you want to start a never-ending religion war, you just bring these arguments in some news group or something like that. And also, if you go to object-oriented programming, you get craziness. Any time of convention, those are just two that I experienced. So peer review is not really a good solution. We need some way to measure in some, let's say, fairly objective way. We don't think that our solution is the ultimate solution, but something better than peer review. And the idea is, if you want to measure, for example, how easy is to debug a code and how robust is a code with respect to error-programming errors? Well, let's just try it. So this is our recipe. Take a handful of programmers. Have the programmer write some code. Then take a handful of so-called spoilers. Give the code to the spoilers. And the spoilers will, their objective is to put some nasty subtle, maybe randomly appearing bugs in the code. Then you get some buggy codes. You take a handful of counter spoilers, give the buggy code them, let Marine and measure how much time the counter spoilers need to debug the code. At the end of the day, you have a data set where you have a spoiled program, counter spoiler, and debugging time. Intuitively, the debugging time is a function of the quality of the original program. If you write in, for example, obfuscated C, that's very easy to put a bug in it. And the quality of the counter spoiler, and on the other plate of the scale, you have the spoiler and the spoiling weight. The spoiling weight is a measure of the amount of code changed by the spoiler. Of course, if you let the spoiler change completely the code, it's too easy to put bugs in it. In order to take an account that the more you write, the easier it is to put a bug, we take an account also of this spoiling weight. We are still working on what really spoiling weight means formally, but more or less it will be the number of lane changes, something like that. So in the end, we have a set of times that depends on those quantities, and we want to split the data we have into the quantities. And for that, we need a model. Basically, we say that the bug in time is a function of the score of the program, of the score of the spoiler, and score of the counter spoiler, plus the spoiling weight, plus some random noise, because of course it depends on maybe many other variables that we cannot handle. The idea is that we have a set of equations, one equation for each counter spoiler program, and we have this number of nodes, this is number of programmers, the number of spoilers, and number of counter spoilers, and if you get enough data, we can solve the previous set of equations, maybe by this square or something like that. Those are just the things that we need to fix, but the details. At the end of the day, we have a score for everyone, so we can give prizes to everyone. We reward the best programmer, the best spoiler, and the best counter spoiler. So the idea of giving prizes to everyone is that in this way the spoiler and the counter spoilers have some motivation to try to do the best work as possible. This approach is about code robustness and easy to debug stuff like that, but if you want, for example, to reward easy of changing the code, you can do something similar. This is the Bell and Whistel competition. You have the programmers that write the code. The code is given to, let's still call them counter spoilers for commodity, but to those counter spoilers, and to the counter spoilers, we say, oh, add this and this new feature to the code, and you measure the amount of time required to add the feature. And still, the amount of time is a function of how well was written the code, how good the counter spoiler is, and how difficult it is to add the new feature. And basically the idea, I don't understand. So how much time do I have? OK. What now? As I told at the beginning of my talk, we are still working on this. We are still designing a web interface called submission and stuff like that. We are still thinking about the score model, the function that I showed you just like before, and we are going to do some experiments. The first few competitions will be mostly experimental because we need to see what it works and what it does. And if you want to keep in touch, I open a LinkedIn group. There is this website. It's a Google site. I just set up two days ago, not kidding, together with the LinkedIn group. And if you want, in the first slide, where is home, yeah, in the first slide, there is my email address. So if you want, feel free to write me. And I guess that this is all the user's slide of conclusion that we, usually, academic people do. So we have two minutes left before the time expires. No, we are still working on that. We... Sorry? Okay, he asked me that if we already did this competition yet, not yet. We just started working on it. And we thought that this occasion was a good occasion to make some, to let people know about this. We hope to be able to do it during this year. Okay. Okay. The slide with the website. Sites Google counterspalling.
 Thank you. Good afternoon to everyone.  I am Ricardo Bernardini.  I work at the University of Udine and I guess you can smell  my dressing code and my slides will smell like academic,  but never mind.  I'm here to talk a little bit about  an idea that we got at the University of Udine to  organize a competition that rewards software  maintainability and easy of reading and stuff like that.  So, a brief outline.  First, what is my objective with this talk.  Then, what is the problem?  What is the solution?  What we are doing right now?  And some conclusion.  This is very academic.  So, my objective is to tell you about this idea about how to  organize competition that rewards software quality.  And why do I want to do that?  Because first of all, I guess it's something interesting to share.  Also, we are still working on this stuff, so we are open to  feedback.  We will be very happy to hear feedback from you.  And also, of course, we are also searching for competitors  because a competition without competitors is not going anywhere.  So, the problem.  Our idea was, you know, usually people put a strong emphasis  on quality, software quality is like efficiency or maybe how  fast you can code.  There is this well-known, actively extreme programming  competition.  Or maybe also ingenuity of solution or stuff like that.  But there are not many competition that rewards other  types of software that, in our opinion, are also very important.  This is how good a software is written.  How easy is to maintain, to read, to modify?  I have some experience in modifying or pursuing code, both to  correct maybe minor bugs.  I have some times to modify them for my work.  I work in digital signal processing.  And I have experience with, for example, a very well-known  program that I wanted to modify for using my research.  And the code is so confusing that in the end I gave up.  So, we decided to try to create a competition that rewards  also this kind of quality.  Those kind of competition are not common.  And there is a very good way, reason.  It's easy to measure how fast you can code.  I say code a shell sort algorithm.  Start, stop, 20 minutes.  Nice.  That's easy.  It's easy to measure how efficient the program is.  Start, stop.  That's nice.  But how can you measure something like how easy is to  read the program?  How easy is to change to debug it?  The first solution that comes to mind is peer review.  Having someone look at the code and say, oh, this is a very  well-written, this is not so well-written, so.  But, unfortunately, peer review is quite subjective.  And people have usually very strong opinion about  coding styles.  For example, Hungarian notation.  Just let me do a check.  How many of you do know what Hungarian notation is?  OK.  Very good.  How many of you do love it?  Do you like it?  A few hands.  OK.  I guess the other one just hated it.  OK.  I'm going to the second group, by the way.  Casing convention.  Do you prefer underscores?  Cable convention, mixed convention.  And also there is the so-called section title convention that  makes uppercase, nonce, and verbs, lowercase, everything else.  Code blocks.  How do you write blocks?  This way, this way, with a large indent, small indent, using  tab, using spaces.  If you want to start a never-ending religion war, you  just bring these arguments in some news group or something  like that.  And also, if you go to object-oriented programming, you  get craziness.  Any time of convention, those are just two that I experienced.  So peer review is not really a good solution.  We need some way to measure in some, let's say, fairly objective  way.  We don't think that our solution is the ultimate solution,  but something better than peer review.  And the idea is, if you want to measure, for example, how easy  is to debug a code and how robust is a code with respect to  error-programming errors?  Well, let's just try it.  So this is our recipe.  Take a handful of programmers.  Have the programmer write some code.  Then take a handful of so-called spoilers.  Give the code to the spoilers.  And the spoilers will, their objective is to put some nasty  subtle, maybe randomly appearing bugs in the code.  Then you get some buggy codes.  You take a handful of counter spoilers, give the buggy code  them, let Marine and measure how much time the counter spoilers  need to debug the code.  At the end of the day, you have a data set where you have  a spoiled program, counter spoiler, and debugging time.  Intuitively, the debugging time is a function of the quality  of the original program.  If you write in, for example, obfuscated C, that's very easy  to put a bug in it.  And the quality of the counter spoiler, and on the other  plate of the scale, you have the spoiler and the spoiling  weight.  The spoiling weight is a measure of the amount of code  changed by the spoiler.  Of course, if you let the spoiler change completely the code,  it's too easy to put bugs in it.  In order to take an account that the more you write, the  easier it is to put a bug, we take an account also of this  spoiling weight.  We are still working on what really spoiling weight means  formally, but more or less it will be the number of lane  changes, something like that.  So in the end, we have a set of times that depends on those  quantities, and we want to split the data we have into the  quantities.  And for that, we need a model.  Basically, we say that the bug in time is a function of the  score of the program, of the score of the spoiler, and  score of the counter spoiler, plus the spoiling weight, plus  some random noise, because of course it depends on maybe  many other variables that we cannot handle.  The idea is that we have a set of equations, one equation for  each counter spoiler program, and we have this number of  nodes, this is number of programmers, the number of  spoilers, and number of counter spoilers, and if you get  enough data, we can solve the previous set of equations,  maybe by this square or something like that.  Those are just the things that we need to fix, but the  details.  At the end of the day, we have a score for everyone, so we  can give prizes to everyone.  We reward the best programmer, the best spoiler, and the  best counter spoiler.  So the idea of giving prizes to everyone is that in this way  the spoiler and the counter spoilers have some motivation  to try to do the best work as possible.  This approach is about code robustness and easy to debug  stuff like that, but if you want, for example, to reward  easy of changing the code, you can do something similar.  This is the Bell and Whistel competition.  You have the programmers that write the code.  The code is given to, let's still call them counter spoilers  for commodity, but to those counter spoilers, and to the  counter spoilers, we say, oh, add this and this new feature  to the code, and you measure the amount of time required  to add the feature.  And still, the amount of time is a function of how well  was written the code, how good the counter spoiler is, and  how difficult it is to add the new feature.  And basically the idea, I don't understand.  So how much time do I have?  OK.  What now?  As I told at the beginning of my talk, we are still working  on this.  We are still designing a web interface called submission  and stuff like that.  We are still thinking about the score model, the function  that I showed you just like before, and we are going to do  some experiments.  The first few competitions will be mostly experimental  because we need to see what it works and what it does.  And if you want to keep in touch, I open a LinkedIn group.  There is this website.  It's a Google site.  I just set up two days ago, not kidding, together with the  LinkedIn group.  And if you want, in the first slide, where is home, yeah,  in the first slide, there is my email address.  So if you want, feel free to write me.  And I guess that this is all the user's slide of conclusion  that we, usually, academic people do.  So we have two minutes left before the time expires.  No, we are still working on that.  We...  Sorry?  Okay, he asked me that if we already did this competition  yet, not yet.  We just started working on it.  And we thought that this occasion was a good occasion to  make some, to let people know about this.  We hope to be able to do it during this year.  Okay.  Okay.  The slide with the website.  Sites Google counterspalling.
Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_0_59.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_60_4523.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_4524_5063.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_5064_5843.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_5844_7247.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_7248_7955.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_7956_8723.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_8724_9707.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_9708_13463.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_13464_19643.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_19644_19823.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_19824_19991.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_19992_20111.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_20112_20891.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_20892_20975.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_20976_21071.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_21072_21215.png Spoiling_and_Counter-spoiling_x-aLE5kz0PU_2_21216_21277.png