The Art of Performance Evaluation
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Hayamizu, Yuto
"Contrary to common belief, performance evaluation is an art." (Raj Jain, 1991) Successful performance evaluation may not be achieved with merely executing common benchmarking tools. This talk presents fundamental principles of performance evaluation and how you can put them into practice. Do you understand what exactly "pgbench" does? Is it appropriate workload for your performance evaluation goal? Common benchmarking tools like "pgbench" are handy for just comparing system A and system B, but if you intend to deeply understand the performance of your system, answers to these questions are critical. In order to conduct a meaningful performance evaluation, the methodology should be elegantly designed to meet the goal of the evaluation: choose metrics for the goal, and choose observation techniques for the metrics. Each step requires careful consideration and deep knowledge about the target system. It cannot be done mechanically. This is why performance evaluation is an art. This talk presents principles of designing performance evaluations and shows how you can put them into practice by introducing the speaker's experiences of performance evaluations with PostgreSQL.
 So today I'm going to talk about the art of performance evaluation. I'm a university researcher at the University of Tokyo and I'm doing database and storage system research. So this talk is going to be a little bit conceptual and opinionated talk so I hope you could enjoy this talk and I'm happy if I can hear your opinion on this topic. So let's get started with this question. What is the role of information technology? What is the role of information technology? My answer is the role of information technology is transforming the computing power into business or social values or you can just say users value. This is a pavulating machine, the first automatic data processing machine in the world. This is built for national population census in the United States. So population is the very fundamental information for the nation. And actually this pavulating machine is converting machine and empower into valuable information. And performance is the key criteria of this transformation efficiency. So computing power is not necessarily machine. Actually human is a kind of computing power so national population census was done by human historically and pavulating machine drastically improved the performance of data processing. So the evolution of information technology is actually the history of this improvement of this transformation. And today technology trend is still basically about transformation of computing power into users value. Big data is a very, very good example. This is an attempt to extract valuable information from massive amount of data. Or internet of things or cyber physical system. This is an attempt to capture the real world with many senses and doing something interesting. So today the data is the source of values and the performance of data processing is becoming more and more important. So from technical perspective, database system is the very important fundamental of transforming data processing power into users value. So database engineers are very important today because they know how this works. So their deep knowledge of database system is directly connected to the performance of system. And performance study of database system is really end to end. Maybe you have to know how hard disk works, how SSD works, how processor works, how memory works, how network works. And of course you have to know how application works and even how business works. Okay, so it's really end to end from physics of hardware to logic of application. And every component technology is changing and growing very rapidly in this field of IT. And sometimes architecture of computing system is also drastically changes. So learning every component technology is necessary but not sufficient. So important point is, oh sorry, important point is acquire the way of learning how this transformation works, how database system can get work together and achieve performance. That is performance evaluation. Performance evaluation is the key process to understand performance of systems. And the important point is, performance evaluation must be users value-oriented. So every performance evaluation starts with setting a goal which contributes to the user's value. And then evaluation procedure is designed and designed toward such goal. When talking about performance evaluation, maybe many people think performance evaluation, oh it's benchmarking. I can do it with Unix Bench or PG Bench or DBT3. Yeah, that's all. Or you may remind some blog posts, our fresh new no secret is faster than your old fresh and classic secret database. Something like that. But frankly speaking, these are not performance evaluation. Yeah, benchmarking is important skills and it's actually the part of performance evaluation. But this is just a mean of performance evaluation. And there are many individual skills which is necessary for performance evaluation. But they are not the core part of performance evaluation. So performance evaluation is goal-oriented process. So the philosophy of orchestrating individual skills is the core part of performance evaluation. Let's take an example story. For example, if your boss says, we need faster storage for our post-wrest database. So maybe your goal might be find the best price for performance SSDs. And one possible approach is list all SSDs available on the market and buy all SSDs available on the market. And try all SSDs with PG Bench. And you can find best SSD. Good? This brute force style makes sense if you have much, much money and much, much time. But if you have a philosophy of performance evaluation, you can do it smarter. First, characterize the workload and model performance with some performance metrics and estimate the performance. So if you have model, you can estimate performance with specs on data sheets. And so you can validate your model with measurement of several SSDs. So in this point, you can actually measure the performance with actual SSDs. And so that validation confirms the model. And you have now a good precision model and now down the candidate SSDs. And finally, confirm the detailed performance on candidate SSDs and you get the final result. So good point of this approach is you do not have to try all SSDs on the market. And acquire model can be used in the future evaluation. That's good. And another important aspect of performance evaluation is that goal could be changed. Because performance evaluation is the process of understanding performance, so its progress might deepen the insight into the target system. And sometimes it identifies more important goals and goal will be changed. So key point is keep considering what is the important goal for your user's value. And then when goal is changed, then orchestrate the individual skills towards the goal. So this is the philosophy of performance evaluation. And it enables you to meaningful performance evaluation. Or this philosophy can be said as the art of performance evaluation. This is very famous book about performance evaluation. And it said performance evaluation is an art. So it cannot produce mechanically and it requires intimate knowledge of target system and careful consideration about selection of methods and tools. So how can we develop our own philosophy? This is not easy, honestly speaking. And a cornerstone of philosophy is experience a lot. But important thing is the theory behind experiences. So theory forms the foundation of experiences. Experiences helps deepening the understanding of the theory. So keep trying this cycle. This is the way to develop your own philosophy. And so as a database researcher, I experienced various performance evaluations. So let me introduce some examples. This is the measurements of latency of processor cache and memory system. In NUMA architecture, each processor has local memory and processors are connected with interconnect like QPI. So access to remote memory takes extra cycles, extra CPU cycles. Or modern processor has multi-level cache. So smaller cache is faster like this. So it is a little bit difficult to measure this latency, but it is possible. So maybe this is an interesting graph for you. Or another example is the case with traditional disk array. This disk array is mid-range class enterprise storage with 160 hard disk drives. And so when I start my master course, I did many measurements on this machine. For example, IOPS passes response time like that. And more interesting example is performance of PCI Express Flash Storage. So PCI Express Flash Storage, IO latency is very, very low relative to existing storage devices. And so in this case, latency of processor interconnect is very visible as a portion of IO latency. You can measure that. Another study is power consumption and energy saving of database system. So this is my master thesis. So modern processor has functionality of changing operating frequency in the runtime. And so by injecting application performance information into frequency controller, aggressive energy saving can be achieved with maintaining some service levels. And experience sometimes requires collection of hard work. So this is the graph of power consumption fluctuation. So even if the system is idle, power consumption is fluctuated by many factors like temperature or total power load in the building, blah, blah, blah, something like that. In order to measure power consumption precisely, I had to do experiments, including the power is very stable period. So this graph shows power consumption is relatively stable from 0AM to 6AM. So when I was doing this experiment, I showed up laboratory every midnight and start experiment at 0AM and finished experiment at 6AM and went to bed when everyone is waking up. This is just a side story. So the remaining part of this talk, I present the fundamentals of performance evaluation for principles and its basic techniques. This is just a basic theory because I don't have much time. So if you want to learn more detail, you can try this book. And in this talk, I'm going to explain the theory with my practice and experiences. Okay, let's move on to principles. Principles are very simple. As I repeated, define a goal first. So performance evaluation must be user-value-oriented. So define a goal which contributes to user's value. And this is very, very difficult process usually. So initial question of performance problem might be very value and usually subjective. So your job is to elaborate it, elaborate them and set a clear and objective goal. So if you can set a goal, maybe you already have finished 50% of your job or 60% of your job. So the remaining process keeps the goal oriented and design the procedure towards the goal using appropriate techniques. Okay, so move on to basic techniques of performance evaluation. There are three basic techniques of performance evaluation. And first one is modeling. Modeling is expressing the performance in a quantitative form, usually in mathematical form. The real system is usually too complex to analyze every single detail. So approximation is the key to understand the mechanism of a target system and find the important variables and formulate the performance in a mathematical form. So this is modeling on the core of scientific approach. Okay, let me give you an example. So maybe you know how hard disk works and the hard disk consists of rotating product and the thick header, the magnetic header to read data on product. So the throughput of hard disk drives can be modeled by this easy mathematical formula. So this is how many bits are swept by the head in a second. So from this model, you can know that throughput is proportional to the density of cylinder. And the density of outermost cylinder is the largest. So inner cylinder has lower density. And you can confirm that with measurements. So measurements clearly depict the performance. So the outermost product has the largest throughput and the inner most in a product has lower throughput. Another example, the latency of hard disk drives. So latency of hard disk drives consists of thick latency and rotational latency. So if you are lucky enough, the rotation of latency is almost zero. But in the most unlucky case, you have to wait for the product to go round. So latency can be expressed in this form. And you can confirm the model by measurement. This graph shows the discolour latency and x-axis shows the thick distance and y-axis shows the latency. So as you can see, the larger thick distance increase latency and also this width of latency band is exactly equals to the rotational latency of this hard disk drive over a 10K RPM. And another popular modeling technique, queuing model. Queuing model consists of a server and a waiting queue. And its behavior is described by a rival rate of customer and average service time. So when you consider CPU performance, you can model CPU as a customer. CPU instructions and service is execution unit in CPU. So when you consider storage system, customer is IO request and service is storage controller or individual disks. And when you consider database system, customer is QLE and service is QLE executor process. And the simplest and useful modeling is MM1. I don't explain too much detail, but this consists of one service and one infinite queue. And with some conditions on a rival rate and service time distribution. And if you do a simple math, average response time can be calculated as this simple formula. And it is not so difficult to extend this to multiple servers. And using this model, we can do some interesting analysis. So here I present performance model of OLTP system with variable operating processor frequency. So frequency goes down, the throughput decreases, and response time increases. So you can model the response time with multiple service queueing model. And so by parameterizing processor frequency, I can formulate the average response time in this way. And this graph shows the curve of model average response time. So y-axis is the throughput and x-axis is throughput of system. And the measurement results is this graph. So as you can see, this model clearly matches with the trend of actual measurement. So even with this simple model, you can evaluate the performance impact of frequency. So this measurement was the basis of application performance-aware energy saving, as I explained in the format slide. So modeling is a very important step to understand system. Maybe good model provides you a good view and enables you to predict performance and scaling trends. And also it helps you to notice that incorrect measurement value caused by implementation bugs because you know the trend of performance by model. But the system is actually very complex usually. Sometimes modeling can result in a bad model. But there is not bad news. Bad model is also informative because it indicates that you are missing something important in modeling or measurements. So you can try to improve your model. And maybe it also indicates that system is too complex. So in that case, this is a sign that you should change the approach toward the goal. So next technique is measurement. When doing performance evaluation, many people tend to do measurement fast, but measurement without modeling is totally pointless because you cannot understand the results. So if you have a model, you can validate the result with a model so you can understand, oh, this is working. And if there is a difference between measurement and model, you can notice that model or measurement something is wrong. So do not start with measurement and do start with modeling. And this is a guide for meaningful measurement. So measurement is not just running benchmarking tool. There are many things to be considered. So work load and metrics, measurement environments, and measurement methods and tools, then conduct measurement and analyze results. So this is a very simplified example story. So here I assume that I want a story system that can support some TPS transactions throughput. And work load is random IO for many clients. So in this case, metric is IO of story system. So before measurement, model performance. In this case, work size and concurrency of IO might be important for performance. And also the IOQ depth in various layers in system might be important. And design measurement environment before, design carefully measurement environment before conducting measurement. So when you want to measure IO of story system, ensure that IO request is really reaching the story system, not cashed by main memory. And performance measurement is monitoring of flow of resources in the system. So designing environment is designing the flow of resources in the system. And then pick appropriate methods or tools. So I repeat that performance evaluation is a very goal-oriented process. So do not start with your familiar tool but appropriate tool for the goal. And be careful about what metric is really measured. For example, IO study is very famous tool for measuring IO performance. It can measure the number of IO requests issued from operating system, but it cannot measure the number of IO requests actually issued to the individual disks behind the storage controller or the number of lead-write system calls from application. So be careful about what you are really watching. So we are all set and run measurement. And maybe in simple case you can run measurement with exhaustive parameter values. But when you are doing measurement with multiple variables, maybe it is unrealistic. So plan your experiments with minimal steps. And we have a good model. The model will tell you where you should measure next. And after the measurement, do not forget to analyze the result. So if your result matches with the model, this is very good. It means you have a correct understanding of your system. That's very good. But if results do not match with your model, something is wrong. So keep trying to improve model or measurement method. Okay, let's move on to the simulation. Simulation is the... So if you do not have a target system, or if it does not exist in the real world yet, simulation is a very effective way to evaluate performance. Or simulation is also used for detailed behavior analysis. Modeling is usually difficult. Modeling every single detail of the real system is usually difficult. So modeling is done with some approximation. But simulation can do much more detail. And in terms of the database system, one interesting simulation technique is IO replay. In IO replay, first trace the IO pattern during the actual execution of workload, and replay the IO pattern on hypothetical devices. So in this way, you can evaluate the performance of various devices with realistic workload. This graph shows the results of IO replay on various devices with TPCC benchmark, in this case. And so replaying IO workload on various hypothetical system like hard disk and SATA disk and PCHX SSD. So you can observe how IO can be handled in every system in very detail. Simulation is a kind of what if analysis based on some assumptions. So unrealistic assumption can easily lead to nonsense results. So be careful about assumptions and so simulation without other techniques is a bit dangerous. So keep in mind that assumption must be validated with other techniques like modeling or measurement. Okay, now I covered the three basic techniques, modeling, measurement, and simulation. And each technique is not so useful by itself. So in order to deeply understand the system's performance using multiple techniques and validating each other is the key. Okay, so I'm wrapping up this talk. Maybe I have speak a little bit faster than I fixed it. So in this talk I talked about the art of performance evaluation. The art of performance evaluation is very, very fundamental ability or capacity of database engineer. Because it is a key process to understand how system works and how system achieves performance through end to end. So because performance evaluation should contribute to user's value, this is very goal oriented process. And then procedure of performance evaluation should be designed with basic techniques, modeling, measurement, and simulation. So this is the basic theory of performance evaluation. And based on this theory, your experiences may develop your own philosophy of performance evaluation. Okay, that's all for my talk. Thank you for listening. Do you have any questions? Could you present the... That's a very important problem. And so in that point, I'm not so experienced because I'm a university guy. So maybe all of you are more experienced in communicating with the business people. But the important point is, yes, simplifying things as much as possible. So if you... it is possible, so trying to calculate the actual economical impact of introducing new storage systems or something like that. So keep it simple and trying to translate the technical diagram to business person's language. Does it answer your question? Also, in my experience, if you have a... you've got it proven and you've got the results there, so somebody else who's technically going to verify it, take that, keep it that, and then you rephrase it in terms that actually are really helpful to the business unit, such as... like you would suggest an economic impact or impact to users or, importantly, disaster recovery scenarios like business continuity. If you phrase it in terms of those kinds of terms, like, you know, they're going from business kind of data, they'll understand percentile and percentages and stuff like that. So if you're talking about hard impacts, like the bottom line or business continuity or whatever, and then have this all sort of back up saying, this, you know, we've done the research and this is the best way to go, you know. So it sounds like it's also more than art? It is an art. It is a conversation. And yeah, don't overload them with technical jargon because it's going to be a bit of a turn off. I know. You just turned it down on what it is. I think that if you say Google is doing it... And that's why we press out. I think that if you're not specifically at the moment stuck to the business unit, you have no idea what the value is. I still like the comparison between the client having the physical and whether the client can get up and go and walk to the other end and make that distance shorter. And that brings it back to the complex stuff where they see everything. I mean, it's simple. But if that's what you're looking for in terms of control of the stuff you can reach or what you like, this is what I think we can do myself. So speaking metaphorically. Yeah. It's finally there for the likes. And the nice thing about the pilot-controller thing is that it's both one that understands the ambition and what they're doing in this same context. That makes sense. Thank you. Sometimes it's also helpful to add information that is not obviously relevant. Like when we're talking about performance, we are also talking about response times. And if you have human users, they have inherent response time constraints in their nature. If human brain gets distracted when your simple web request takes longer than two seconds, your brain starts looking for alternative things that it can do. So if your computer response time goes beyond certain thresholds, that's no longer tolerable because it impacts the human's performance. And a lot of business people don't see that. They see a cost-benefit between response times and how much that costs them hardware. But how much the productivity loss is when the response time exceeds certain thresholds is important here too, to factor that into cost-benefit analysis. As one example. Any questions? Okay. Thank you so much.
 So today I'm going to talk about the art of performance evaluation. I'm a university researcher at the University of Tokyo and I'm doing database and storage system research.  So this talk is going to be a little bit conceptual and opinionated talk so I hope you could enjoy this talk and I'm happy if I can hear your opinion on this topic.  So let's get started with this question. What is the role of information technology? What is the role of information technology?  My answer is the role of information technology is transforming the computing power into business or social values or you can just say users value.  This is a pavulating machine, the first automatic data processing machine in the world.  This is built for national population census in the United States. So population is the very fundamental information for the nation.  And actually this pavulating machine is converting machine and empower into valuable information.  And performance is the key criteria of this transformation efficiency.  So computing power is not necessarily machine.  Actually human is a kind of computing power so national population census was done by human historically and pavulating machine drastically improved the performance of data processing.  So the evolution of information technology is actually the history of this improvement of this transformation.  And today technology trend is still basically about transformation of computing power into users value.  Big data is a very, very good example. This is an attempt to extract valuable information from massive amount of data.  Or internet of things or cyber physical system. This is an attempt to capture the real world with many senses and doing something interesting.  So today the data is the source of values and the performance of data processing is becoming more and more important.  So from technical perspective, database system is the very important fundamental of transforming data processing power into users value.  So database engineers are very important today because they know how this works. So their deep knowledge of database system is directly connected to the performance of system.  And performance study of database system is really end to end.  Maybe you have to know how hard disk works, how SSD works, how processor works, how memory works, how network works. And of course you have to know how application works and even how business works.  Okay, so it's really end to end from physics of hardware to logic of application. And every component technology is changing and growing very rapidly in this field of IT.  And sometimes architecture of computing system is also drastically changes.  So learning every component technology is necessary but not sufficient.  So important point is, oh sorry, important point is acquire the way of learning how this transformation works, how database system can get work together and achieve performance.  That is performance evaluation. Performance evaluation is the key process to understand performance of systems.  And the important point is, performance evaluation must be users value-oriented.  So every performance evaluation starts with setting a goal which contributes to the user's value.  And then evaluation procedure is designed and designed toward such goal.  When talking about performance evaluation, maybe many people think performance evaluation, oh it's benchmarking.  I can do it with Unix Bench or PG Bench or DBT3. Yeah, that's all. Or you may remind some blog posts, our fresh new no secret is faster than your old fresh and classic secret database.  Something like that. But frankly speaking, these are not performance evaluation.  Yeah, benchmarking is important skills and it's actually the part of performance evaluation.  But this is just a mean of performance evaluation.  And there are many individual skills which is necessary for performance evaluation.  But they are not the core part of performance evaluation.  So performance evaluation is goal-oriented process.  So the philosophy of orchestrating individual skills is the core part of performance evaluation.  Let's take an example story.  For example, if your boss says, we need faster storage for our post-wrest database.  So maybe your goal might be find the best price for performance SSDs.  And one possible approach is list all SSDs available on the market and buy all SSDs available on the market.  And try all SSDs with PG Bench. And you can find best SSD. Good?  This brute force style makes sense if you have much, much money and much, much time.  But if you have a philosophy of performance evaluation, you can do it smarter.  First, characterize the workload and model performance with some performance metrics and estimate the performance.  So if you have model, you can estimate performance with specs on data sheets.  And so you can validate your model with measurement of several SSDs.  So in this point, you can actually measure the performance with actual SSDs.  And so that validation confirms the model. And you have now a good precision model and now down the candidate SSDs.  And finally, confirm the detailed performance on candidate SSDs and you get the final result.  So good point of this approach is you do not have to try all SSDs on the market.  And acquire model can be used in the future evaluation. That's good.  And another important aspect of performance evaluation is that goal could be changed.  Because performance evaluation is the process of understanding performance,  so its progress might deepen the insight into the target system.  And sometimes it identifies more important goals and goal will be changed.  So key point is keep considering what is the important goal for your user's value.  And then when goal is changed, then orchestrate the individual skills towards the goal.  So this is the philosophy of performance evaluation. And it enables you to meaningful performance evaluation.  Or this philosophy can be said as the art of performance evaluation.  This is very famous book about performance evaluation.  And it said performance evaluation is an art.  So it cannot produce mechanically and it requires intimate knowledge of target system  and careful consideration about selection of methods and tools.  So how can we develop our own philosophy?  This is not easy, honestly speaking.  And a cornerstone of philosophy is experience a lot.  But important thing is the theory behind experiences.  So theory forms the foundation of experiences.  Experiences helps deepening the understanding of the theory.  So keep trying this cycle.  This is the way to develop your own philosophy.  And so as a database researcher, I experienced various performance evaluations.  So let me introduce some examples.  This is the measurements of latency of processor cache and memory system.  In NUMA architecture, each processor has local memory and processors are connected with interconnect like QPI.  So access to remote memory takes extra cycles, extra CPU cycles.  Or modern processor has multi-level cache.  So smaller cache is faster like this.  So it is a little bit difficult to measure this latency, but it is possible.  So maybe this is an interesting graph for you.  Or another example is the case with traditional disk array.  This disk array is mid-range class enterprise storage with 160 hard disk drives.  And so when I start my master course, I did many measurements on this machine.  For example, IOPS passes response time like that.  And more interesting example is performance of PCI Express Flash Storage.  So PCI Express Flash Storage, IO latency is very, very low relative to existing storage devices.  And so in this case, latency of processor interconnect is very visible as a portion of IO latency.  You can measure that.  Another study is power consumption and energy saving of database system.  So this is my master thesis.  So modern processor has functionality of changing operating frequency in the runtime.  And so by injecting application performance information into frequency controller,  aggressive energy saving can be achieved with maintaining some service levels.  And experience sometimes requires collection of hard work.  So this is the graph of power consumption fluctuation.  So even if the system is idle, power consumption is fluctuated by many factors like temperature or total power load in the building, blah, blah, blah, something like that.  In order to measure power consumption precisely, I had to do experiments, including the power is very stable period.  So this graph shows power consumption is relatively stable from 0AM to 6AM.  So when I was doing this experiment, I showed up laboratory every midnight and start experiment at 0AM and finished experiment at 6AM  and went to bed when everyone is waking up.  This is just a side story.  So the remaining part of this talk, I present the fundamentals of performance evaluation for principles and its basic techniques.  This is just a basic theory because I don't have much time.  So if you want to learn more detail, you can try this book.  And in this talk, I'm going to explain the theory with my practice and experiences.  Okay, let's move on to principles.  Principles are very simple.  As I repeated, define a goal first.  So performance evaluation must be user-value-oriented.  So define a goal which contributes to user's value.  And this is very, very difficult process usually.  So initial question of performance problem might be very value and usually subjective.  So your job is to elaborate it, elaborate them and set a clear and objective goal.  So if you can set a goal, maybe you already have finished 50% of your job or 60% of your job.  So the remaining process keeps the goal oriented and design the procedure towards the goal using appropriate techniques.  Okay, so move on to basic techniques of performance evaluation.  There are three basic techniques of performance evaluation.  And first one is modeling.  Modeling is expressing the performance in a quantitative form, usually in mathematical form.  The real system is usually too complex to analyze every single detail.  So approximation is the key to understand the mechanism of a target system  and find the important variables and formulate the performance in a mathematical form.  So this is modeling on the core of scientific approach.  Okay, let me give you an example.  So maybe you know how hard disk works and the hard disk consists of rotating product and the thick header,  the magnetic header to read data on product.  So the throughput of hard disk drives can be modeled by this easy mathematical formula.  So this is how many bits are swept by the head in a second.  So from this model, you can know that throughput is proportional to the density of cylinder.  And the density of outermost cylinder is the largest.  So inner cylinder has lower density.  And you can confirm that with measurements.  So measurements clearly depict the performance.  So the outermost product has the largest throughput and the inner most in a product has lower throughput.  Another example, the latency of hard disk drives.  So latency of hard disk drives consists of thick latency and rotational latency.  So if you are lucky enough, the rotation of latency is almost zero.  But in the most unlucky case, you have to wait for the product to go round.  So latency can be expressed in this form.  And you can confirm the model by measurement.  This graph shows the discolour latency and x-axis shows the thick distance and y-axis shows the latency.  So as you can see, the larger thick distance increase latency and also this width of latency band  is exactly equals to the rotational latency of this hard disk drive over a 10K RPM.  And another popular modeling technique, queuing model.  Queuing model consists of a server and a waiting queue.  And its behavior is described by a rival rate of customer and average service time.  So when you consider CPU performance, you can model CPU as a customer.  CPU instructions and service is execution unit in CPU.  So when you consider storage system, customer is IO request and service is storage controller or individual disks.  And when you consider database system, customer is QLE and service is QLE executor process.  And the simplest and useful modeling is MM1.  I don't explain too much detail, but this consists of one service and one infinite queue.  And with some conditions on a rival rate and service time distribution.  And if you do a simple math, average response time can be calculated as this simple formula.  And it is not so difficult to extend this to multiple servers.  And using this model, we can do some interesting analysis.  So here I present performance model of OLTP system with variable operating processor frequency.  So frequency goes down, the throughput decreases, and response time increases.  So you can model the response time with multiple service queueing model.  And so by parameterizing processor frequency, I can formulate the average response time in this way.  And this graph shows the curve of model average response time.  So y-axis is the throughput and x-axis is throughput of system.  And the measurement results is this graph.  So as you can see, this model clearly matches with the trend of actual measurement.  So even with this simple model, you can evaluate the performance impact of frequency.  So this measurement was the basis of application performance-aware energy saving, as I explained in the format slide.  So modeling is a very important step to understand system.  Maybe good model provides you a good view and enables you to predict performance and scaling trends.  And also it helps you to notice that incorrect measurement value caused by implementation bugs  because you know the trend of performance by model.  But the system is actually very complex usually.  Sometimes modeling can result in a bad model.  But there is not bad news. Bad model is also informative because it indicates that you are missing something important in modeling or measurements.  So you can try to improve your model.  And maybe it also indicates that system is too complex. So in that case, this is a sign that you should change the approach toward the goal.  So next technique is measurement.  When doing performance evaluation, many people tend to do measurement fast, but measurement without modeling is totally pointless because you cannot understand the results.  So if you have a model, you can validate the result with a model so you can understand, oh, this is working.  And if there is a difference between measurement and model, you can notice that model or measurement something is wrong.  So do not start with measurement and do start with modeling.  And this is a guide for meaningful measurement.  So measurement is not just running benchmarking tool.  There are many things to be considered.  So work load and metrics, measurement environments, and measurement methods and tools, then conduct measurement and analyze results.  So this is a very simplified example story.  So here I assume that I want a story system that can support some TPS transactions throughput.  And work load is random IO for many clients.  So in this case, metric is IO of story system.  So before measurement, model performance.  In this case, work size and concurrency of IO might be important for performance.  And also the IOQ depth in various layers in system might be important.  And design measurement environment before, design carefully measurement environment before conducting measurement.  So when you want to measure IO of story system, ensure that IO request is really reaching the story system, not cashed by main memory.  And performance measurement is monitoring of flow of resources in the system.  So designing environment is designing the flow of resources in the system.  And then pick appropriate methods or tools.  So I repeat that performance evaluation is a very goal-oriented process.  So do not start with your familiar tool but appropriate tool for the goal.  And be careful about what metric is really measured.  For example, IO study is very famous tool for measuring IO performance.  It can measure the number of IO requests issued from operating system,  but it cannot measure the number of IO requests actually issued to the individual disks behind the storage controller  or the number of lead-write system calls from application.  So be careful about what you are really watching.  So we are all set and run measurement.  And maybe in simple case you can run measurement with exhaustive parameter values.  But when you are doing measurement with multiple variables, maybe it is unrealistic.  So plan your experiments with minimal steps.  And we have a good model. The model will tell you where you should measure next.  And after the measurement, do not forget to analyze the result.  So if your result matches with the model, this is very good.  It means you have a correct understanding of your system.  That's very good.  But if results do not match with your model, something is wrong.  So keep trying to improve model or measurement method.  Okay, let's move on to the simulation.  Simulation is the...  So if you do not have a target system, or if it does not exist in the real world yet,  simulation is a very effective way to evaluate performance.  Or simulation is also used for detailed behavior analysis.  Modeling is usually difficult.  Modeling every single detail of the real system is usually difficult.  So modeling is done with some approximation.  But simulation can do much more detail.  And in terms of the database system, one interesting simulation technique is IO replay.  In IO replay, first trace the IO pattern during the actual execution of workload,  and replay the IO pattern on hypothetical devices.  So in this way, you can evaluate the performance of various devices with realistic workload.  This graph shows the results of IO replay on various devices with TPCC benchmark, in this case.  And so replaying IO workload on various hypothetical system like hard disk and SATA disk and PCHX SSD.  So you can observe how IO can be handled in every system in very detail.  Simulation is a kind of what if analysis based on some assumptions.  So unrealistic assumption can easily lead to nonsense results.  So be careful about assumptions and so simulation without other techniques is a bit dangerous.  So keep in mind that assumption must be validated with other techniques like modeling or measurement.  Okay, now I covered the three basic techniques, modeling, measurement, and simulation.  And each technique is not so useful by itself.  So in order to deeply understand the system's performance using multiple techniques and validating each other is the key.  Okay, so I'm wrapping up this talk.  Maybe I have speak a little bit faster than I fixed it.  So in this talk I talked about the art of performance evaluation.  The art of performance evaluation is very, very fundamental ability or capacity of database engineer.  Because it is a key process to understand how system works and how system achieves performance through end to end.  So because performance evaluation should contribute to user's value, this is very goal oriented process.  And then procedure of performance evaluation should be designed with basic techniques, modeling, measurement, and simulation.  So this is the basic theory of performance evaluation.  And based on this theory, your experiences may develop your own philosophy of performance evaluation.  Okay, that's all for my talk. Thank you for listening.  Do you have any questions?  Could you present the...  That's a very important problem.  And so in that point, I'm not so experienced because I'm a university guy.  So maybe all of you are more experienced in communicating with the business people.  But the important point is, yes, simplifying things as much as possible.  So if you... it is possible, so trying to calculate the actual economical impact of introducing new storage systems or something like that.  So keep it simple and trying to translate the technical diagram to business person's language.  Does it answer your question?  Also, in my experience, if you have a... you've got it proven and you've got the results there, so somebody else who's technically going to verify it,  take that, keep it that, and then you rephrase it in terms that actually are really helpful to the business unit, such as...  like you would suggest an economic impact or impact to users or, importantly, disaster recovery scenarios like business continuity.  If you phrase it in terms of those kinds of terms, like, you know, they're going from business kind of data, they'll understand percentile and percentages and stuff like that.  So if you're talking about hard impacts, like the bottom line or business continuity or whatever, and then have this all sort of back up saying,  this, you know, we've done the research and this is the best way to go, you know.  So it sounds like it's also more than art?  It is an art. It is a conversation.  And yeah, don't overload them with technical jargon because it's going to be a bit of a turn off.  I know.  You just turned it down on what it is.  I think that if you say Google is doing it...  And that's why we press out.  I think that if you're not specifically at the moment stuck to the business unit, you have no idea what the value is.  I still like the comparison between the client having the physical and whether the client can get up and go and walk to the other end and make that distance shorter.  And that brings it back to the complex stuff where they see everything.  I mean, it's simple.  But if that's what you're looking for in terms of control of the stuff you can reach or what you like, this is what I think we can do myself.  So speaking metaphorically.  Yeah. It's finally there for the likes.  And the nice thing about the pilot-controller thing is that it's both one that understands the ambition and what they're doing in this same context.  That makes sense.  Thank you.  Sometimes it's also helpful to add information that is not obviously relevant.  Like when we're talking about performance, we are also talking about response times.  And if you have human users, they have inherent response time constraints in their nature.  If human brain gets distracted when your simple web request takes longer than two seconds, your brain starts looking for alternative things that it can do.  So if your computer response time goes beyond certain thresholds, that's no longer tolerable because it impacts the human's performance.  And a lot of business people don't see that.  They see a cost-benefit between response times and how much that costs them hardware.  But how much the productivity loss is when the response time exceeds certain thresholds is important here too,  to factor that into cost-benefit analysis.  As one example.  Any questions?  Okay.  Thank you so much.
The_Art_of_Performance_Evaluation_0_815.png The_Art_of_Performance_Evaluation_816_887.png The_Art_of_Performance_Evaluation_888_1007.png The_Art_of_Performance_Evaluation_1008_1055.png The_Art_of_Performance_Evaluation_1056_3107.png The_Art_of_Performance_Evaluation_3108_4235.png The_Art_of_Performance_Evaluation_4236_6347.png The_Art_of_Performance_Evaluation_6348_6707.png The_Art_of_Performance_Evaluation_6708_7595.png The_Art_of_Performance_Evaluation_7596_8531.png The_Art_of_Performance_Evaluation_8532_8603.png The_Art_of_Performance_Evaluation_8604_8867.png The_Art_of_Performance_Evaluation_8868_9299.png The_Art_of_Performance_Evaluation_9300_9527.png The_Art_of_Performance_Evaluation_9528_12539.png The_Art_of_Performance_Evaluation_12540_13943.png The_Art_of_Performance_Evaluation_13944_14111.png The_Art_of_Performance_Evaluation_14112_14195.png The_Art_of_Performance_Evaluation_14196_14267.png The_Art_of_Performance_Evaluation_14268_14519.png The_Art_of_Performance_Evaluation_14520_14567.png The_Art_of_Performance_Evaluation_14568_14639.png The_Art_of_Performance_Evaluation_14640_14687.png The_Art_of_Performance_Evaluation_14688_15719.png The_Art_of_Performance_Evaluation_15720_17207.png The_Art_of_Performance_Evaluation_17208_17951.png The_Art_of_Performance_Evaluation_17952_18707.png The_Art_of_Performance_Evaluation_18708_19643.png The_Art_of_Performance_Evaluation_19644_22007.png The_Art_of_Performance_Evaluation_22008_24203.png The_Art_of_Performance_Evaluation_24204_25223.png The_Art_of_Performance_Evaluation_25224_26651.png The_Art_of_Performance_Evaluation_26652_27203.png The_Art_of_Performance_Evaluation_27204_28067.png The_Art_of_Performance_Evaluation_28068_29027.png The_Art_of_Performance_Evaluation_29028_30419.png The_Art_of_Performance_Evaluation_30420_31199.png The_Art_of_Performance_Evaluation_31200_33707.png The_Art_of_Performance_Evaluation_33708_35855.png The_Art_of_Performance_Evaluation_35856_35975.png The_Art_of_Performance_Evaluation_35976_37043.png The_Art_of_Performance_Evaluation_37044_37295.png The_Art_of_Performance_Evaluation_37296_37859.png The_Art_of_Performance_Evaluation_37860_38699.png The_Art_of_Performance_Evaluation_38700_38807.png The_Art_of_Performance_Evaluation_38808_38891.png The_Art_of_Performance_Evaluation_38892_41135.png The_Art_of_Performance_Evaluation_41136_41879.png The_Art_of_Performance_Evaluation_41880_42671.png The_Art_of_Performance_Evaluation_42672_42791.png The_Art_of_Performance_Evaluation_42792_43835.png The_Art_of_Performance_Evaluation_43836_44591.png The_Art_of_Performance_Evaluation_44592_45371.png The_Art_of_Performance_Evaluation_45372_46187.png The_Art_of_Performance_Evaluation_46188_46799.png The_Art_of_Performance_Evaluation_46800_58146.png