Advanced Pandas
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Van den Bossche, Joris
Pandas is nowadays the library of choice for manipulating and analysing structured data, providing high-performance, easy-to-use data structures and data analysis tools. In this hands-on tutorial on using pandas for data analysis, you will be guided through some of the powerful methods and concepts in pandas, including time series manipulation (resampling and rolling operations), groupby operations, reshaping with stack/unstack/pivot, … Basic familiarity with pandas data structures (Series/DataFrame) is assumed.
 So the material is going around, and there you can also download it. There is a bunch of material in there. It's a panestero for a full day. We are going to do the notebooks five and six. Probably seven we won't have time for, but there is still some additional material on similar topics as the five and six. So maybe just to introduce, I'm Joris Van Bosse. I'm currently working at INRIA in Paris for the Paris Cyclist Center for Data Science. So this is an advanced tutorial. Advanced panesthetics is always a bit difficult to choose because there are many and I just choose some time series data and some reshaping. Of course, it depends on what you already did with pandas if that will be advanced for you or not. So it's maybe for some people it will be too easy for other two difficult. But there are a lot of exercises in the notebook. The solutions, you can also find them in the repo. I will directly show it. So if I go too slow for you, you can always advance and see the solution yourself. So that's certainly possible. For the rest, I like to do hands-on and that you just try it out yourself. The exercise is just right and I'm here so ask questions. If there is something not clear, if there is something you don't understand, ask as many questions as you can. So we're going to start with the number five time series data. So if you look in the, if you open the Jupyter notebook inside the directory of the tutorial, you will see here there is a number five time series data. So it's that one that you have to open. So, as you will see, the questions are indicated by green boxes. So that means that I would like you to try it yourself. If you tried it, you can always check with the provided solution by uncommenting this cell and running it. And then you will see a possible solution. But I advise you to first try yourself before looking at it. So the first of the two topics I want to talk about are time series data with pandas. So it's time series data is something that I find very, yeah, pandas very useful for and easy for to use. I will try to show some of the features that are available in pandas when you have time series data. So of course, I assume that everybody knows this in the standard library of Python, you have the daytime objects. Just as a reminder, you can construct a daytime like that. You can print it or format it as you want. In pandas, we have our own custom daytime objects. We're subclass from the standard library one. So in principle, everything that you know about the standard daytime object, its attributes, its methods should work on the pandas object as well, but with some additional features in pandas. So the daytime is called timestamp. You can construct it like this from a string. As with the daytime object, you can also pause the individual year, month, day, et cetera, but you can also just construct it from a string. So again, you can access certain attributes. For example, I want to get the month of my timestamp. And apart from that, we also have a time delta object which is similar to the daytime.time delta in the standard library. And with that, you can, for example, add a few days. Now, of course, typically, you won't construct your times yourself by typing the string. You parse them from somewhere. And because there are, unfortunately, many different formats that people use, we have also a quite flexible or powerful method to parse strings that represent day times. And this method is called two-day time. You can also read the CSV file and let a column with daytime data be parsed while reading, but on the root, it will also just use that. So the two-day time function is the workhorse to parse strings. Like this, you can parse a clear format, an easy standard format, and then it will parse it correctly. Of course, once you start doing other things like the 9th of December like that, you will see that this is parsed incorrectly if for European. For somebody from America, it might be correct. But first, it seems wrong. So there is one thing you can do. You can use day first is true to indicate I want to have it parsed assuming that the day comes first and then only the month. And that's maybe the main message here is if you have such custom formats, always it's much better to just specify the exact format on which it will be parsed, like that. If there's somebody who still needs the material, the stick is there. And afterwards, there is somebody there as well. So very basic two-day time we can use for that. But now once you have your data inside a data frame or a series, we can work with those data. For example, I constructed here a series of strings. I'm not sure what happened here. I think this comma shouldn't be there. Apparently, it's OK for two-day time in the comma. But typically, I don't think that was meant like that. So we have a series of strings. Again, like I showed before, with single strings, we can parse it to day time. And now we have timestamp data. You see the difference? The day type here is object. Object means just any Python object possible. But in Pandas, this is also used for strings. Because we don't have a specific data type for strings, we use object. For the second one, now you see this is day time 64. So that's a non-py data type to represent day times. So once we have a series with a correct data type, we get some additional features. Like we could access on the single timestamp object. For example, in our or the month attribute as we did before, we can now do that for the full time series at once. But they are not directly on the series available, but through.dt attributes. For example, with.dt,.hour, I get the hour of each entry of my series. I can get the weekday. And there are many more. If you do a time completion, you will see you can also get the day of the year, the hour, the minute, or you can convert it. The weekday name, which week of the year, et cetera. A lot of attributes are available. If you just want to construct some regular time data, you can also use the date range function to create some data. Now some additional features, it's already useful to have it in a column. But if you have time series data, when you have those data in an index, there are even more additional features which become available in pandas. And from now on, we will use a small example, a data set. It's read it with read CSV. If I look at it, it has one column, the timestamps, and three columns which are measurements of this charge in a certain river in flameless. So it's the flow through the river throughout time. As I showed before, we want to convert our strings to daytime. So we use two daytime. Then I also want to use it as my index, as I just said, so for that I can use.setIndex. I did it multiple times. I got an error because I executed that cell twice. The second time, there was no column time anymore to set as the index. Therefore, it raised an error. So now you can also see it in the representation. Now you see this is my index of my series. I could also have done it directly when parsing the CSV file by indicating that the first column should be my index and that I want to parse those as dates. So if you now look at the index of my data frame, you will see that this is called a daytime index. So if you want to have those features in pandas, make sure that it's a daytime index. And again, just as with a series or a single timestamp, we can access several of those attributes like the year, day of the year, year itself, et cetera. So you see it's data from 2009 up to 2013. So up to now, these are rather the basics of working with date times. My screen is now a bit too zoomed in, apparently. So if we plot it, you will already see that pandas, because it's a time series, it has a daytime index. The axis of the figure is configured. And you can also, if you zoom in, you will see that the formatting of the axis automatically updates to a nice default, as nice looking possible, labels. So up to now, mostly the basics, as I just said. But now I want to show a few things. Once you have... How to select data if you have a time series. So in pandas, we can slice our data, slice the rows by providing here, inside the square brackets, I could do the same actually by using.lock to be more explicit that I want to slice based on the actual labels of my data frame. But you can pass the single labels, for example, I want to select my data from 9 to 7 in the evening on that particular day. So if you do that, you select those specific days. But one additional handy feature is that instead of passing the actual timestamp object, which match our labels, you can also pass it as just a string. Because it's a daytime index, pandas will try to... The slides that you provided will automatically... It knows it has to be dates, so I will try to parse them as dates. Therefore you don't always have to write the full timestamp like that. So this is exactly the same. An additional thing to this, this basic parsing using strings is fine, but there is an additional feature which is a bit more magical is that if you provide a shorter string, for example, 2013, it's interpreted as a slice from the beginning to the end of 2013. So for example, if I execute this, you will see it has selected data from the beginning of 2013 up to the end. You could also say, for example, I want all data of the first of January. And then it's... Again, because it has a higher resolution, it represents multiple rows. It selects all the different hours of that day. It's very useful in sometimes an interactive work, but also more magical to use. So for example... Yeah, so because it's label-based, it's based on actual labels in my time series, then it's inclusive. So if you would slice... I will show it here. If you slice up to 18, the 60-clock, the 18 is included. The 19 was not because there was no 19 there, but so this shows it better that the endpoint is included. So that's something basically with Pandas indexing. You can either index based on actual labels, then the endpoint is included. So if you say I want from 9 to 6, 6 is included. You can also slice by position. I want the first, the second, the third row. In that case, the NumPy rules for indexing apply, and then the endpoint that your supply is not included. Just to quickly show this, if you would do... So you can use the ILOC1 to do positional indexing. So here, this will not select the rows 0, 1, 2, but only 0 and 1, and the 2 is not included. So this gives you two rows instead of including the row at position 2, which is the third row. Okay, some small exercises to try this yourself to select some data of the pool frame. So if you have any problems while trying this out, just raise your hand and ask. Now, if you theta here means I am sitting here, then I have 5 times a veteran sponsored standard, which toward those麵s it affects, if you tune out to 1 large F3 you won't be able to find your direct random strength. I'm not sure why. Can you go up? No, I haven't loaded. No, no, no. But if it doesn't, you can't ignore it if you don't have it. But I suppose you can do it this way with the index directly, right? So without creating an ulceric code. Yeah, maybe I showed both of them. I'm not sure anymore. Yeah, but you need to... Oh, no, I didn't see it. Maybe it's from an older version. Yeah, you don't have to. Yeah, so there I need it. Yeah, but you don't have to. I mean, you can... Because it's serious. Actually, it's a new feature only in the latest. You see? Before, it only works on series, and it's only from the 0.20 release. That also works on the index. So now you can do it directly. Index month is in. But before, it didn't work. Therefore, there was that solution. But I suppose that index.month returns a new series. It returns an index. No, the thing is, it was already... Yeah, I remember, because I actually did a PR for that myself. So the thing is that data.index.month previously returned an empire. Which doesn't have an in-minute, but it changed to return an index. Just the same format index as the original index, an integer index, and then you can use it. So that's the reason that... That's askew of... I didn't succeed in installing Jupyter. Do you have internet? I have internet, but it seems not to be running very fine. I had tried to install it with pip, and it broke off after. And do you know what the error message was? I have to find it again. I think it's another terminal. I have closed it, maybe. And now did you install Python? Python works, that is not a problem. And you have anaconda, but normally it should be included with anaconda, if you try to install anaconda. There is the problem. I have a 32-bit system. I didn't succeed in installing it. Normally, I think they still have downloads for 32-bits. I had tried, I think, and it didn't work. I would try to install the downloaded 32-bits of anaconda. Then everything is available. Because everything here seems to be based on... Yeah. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. The question was, if it's possible to pass a callable to sample, to re-sample. But do you mean the callable to calculate the resulting value based on... I don't have a answer. I have a more specific question. I want to make a result in all of it by a specific function which will be in the top key. I think it's probably then easier to use group by and you can calculate based on our function, you can calculate the value that you want and then that's for. I think it's somehow robust. The reason is weak, the function is very convenient. I use it in any production system. This is something I probably see is rather robust. Yeah. The change over time, I don't know if you have a chance. I think things like group by and resample, as long as you are using the, not too exotic things and for example the building methods on the mean, the median, things like that, normally that should be stable. It's more when you maybe supply your own custom function and there in the, sometimes there are many corner cases where the function can return and things like that, that sometimes things can go wrong or can be fixed or can change. But certainly for the resample and group by, working with those time series, if you use the typical building methods, normally that should be stable. You think when they come up with some time, version 1.0, it should be... But I show here, so the resample syntax has changed some reasons, some versions ago. So that should stay the same as this, as I show it here. For group by, resample, rolling operations. Because resample before was a method where you gave the how to resample and that has changed. But now that should be fine and should be stable for the full panacea.0 series. Okay, before we go to the next notebook, quickly go through some of those exercises. So, yeah, the first one, plot a monthly standard deviation. So we want monthly, so resample by month. Take the standard deviation and we plot it. So here you see a plot of the standard deviation by month. The next one I want to show, because there is an additional thing that can be useful, is I ask for the mean and median value from a single column. You can, of course, do it like this. Yeah, I provide a certain selection here, but I resample and I take the mean, I resample the same and I take the median. But what I can also do is if I take this one out, so this gives me a single column and I resample monthly. But if I want to calculate multiple things of a single column, I can also pass a list to aggregates. And I can say I want the mean and the median, like this, and to calculate both at the same time and you get the data frame with different columns for the different functions, like that. So if you also want to plot this, oops. My mouse. Like that. Because now you have two columns with a name, mean and median, you will also get on the plot on the quick visualization, you also get the legend already there and you don't have to edit yourself. Of course, you might want to edit it out of the default books. So the function that you specify by strain, like mean and median, they're very fast because they're optimized at the C level. So they're on the C. There was a lot of work done with sci-fi by being able to pass C callback functions so that that can bring this down on the river. Is that also going to be included in the address? For now, I don't know of any plans through that. I'm not sure how easy it will be because the... Yeah, the site and implementations of mean for goodbye are very specific for goodbye. Not generic mean functions, but then applied at the site and level on the different groups, but specifically written for goodbye. So I'm not sure how easy that would be, but it's mainly that last part because they are specifically written for goodbye that they are faster. So yeah, I'm not sure about it. Yeah, I will show this one as well. So yeah, plot the monthly, minimum and maximum daily average value of that station. So I first calculate my daily averages and then based on those data, I can do another resample and take of those daily values the monthly, minimum and maximum. And again, I pass a list to the AC function to do to calculate two functions at the same time. And then I get it like that. So a plot with the minimum and the maximum. I want to go to this last one. Maybe it's not available because I only added this on the... just before the tutorial. But an additional exercise which was not included yet. So calculate the typical yearly pattern with a monthly resolution. Because we already saw on the data if I go look at the plot that I showed, if I take here the sum for example, you can already notice each time in the winter, as can be expected, there is more flow, a larger flow rate in the rivers. So here we can calculate monthly values over the different years. But maybe I want to make a plot of how the typical and average yearly profile looks like. So that's the last exercise. And the reason that I want to do added here is because it's also something... if you work with time series data, typical to do for example the daily pattern, yearly pattern. It's not with resample that we will do it, but we can do this with goodbye. So my data frame which was called... yes, data. So instead of resampling every month, which will take every month of every year a separate values, I just want to group by all the months of the different years. So if I take the month attribute of the index, this will give the same value for all the different years for the same month. So if I want to do something similar as resampling, which is also for a certain group, calculates the average or the sum. But now based on those, I can use group by. I can first assign this month to a new column but I can also give just those values to group by. So now we do a group by the one up to 12 of the different months. Take the mean for example like that. So now we get the average for every month, but over the average over the different years. If I plot this one, you will see the typical pattern of the flow rate in the river in the winter month. So you have much higher flow rates as in the summer months. So depending on what kind of averaging you want to do, it's resampled or group by that can be very useful. Okay, are there any remaining questions about time series data? If not, I want to go for the last 35 minutes to the next notebook, which is reshaping data. To go into some of the functions in Panus that you can use to reshape your data. Okay, so reshaping your data is also something that typically you need to do sometimes with your data and there are some different ways to approach this in Panus. I will skip the comparison with Excel for now, but I will just go through this part. So pivot to start with this example. So I made up a small data frame. It's a part of the Titanic data set about the different passengers of the Titanic, whether they survived or not, survived column, whether sex was in which class they traveled, the fare they paid to travel with the Titanic. Now we have columns and for the different variables and every row is one record. So one passenger. You could want to rearrange your data and to show not a single, to not have every row as one record, which is also called your tidy data. Your columns are variables, your rows are records, but sometimes you want to rearrange them to summarize them differently or to present them differently. To only rearrange your data, you can use the pivot method. What do you supply in the pivot method? You say you supply three columns, the column you want as your new index, the column you want as your new column labels, the column for the values to fill in this new data frame. So if I do this, you will see. So now I set the class should be my index. So the one, two, three are now here. My index labels, the columns labels are my female and male and those values in this data frame are the columns with the values from the fair column. So the pivot method in this way, you can rearrange your data. Or another example now with the class and the sex, but the values that I filled in was whether they survived or not. Now, the data frame that I showed here was only a small subset of the actual data. So if I read the full data set of the Titanic, it has not only six values, but if I look at the info, so it has almost 900 entries. So let's suppose that I want to make a similar summary table with the female male as my columns, the class as my index rows, and the fair as my actual values. So if I try this, I get an exception, index contains duplicate entries cannot reshape. It's not the best error message. I have to say so if somebody wants to do a PR to pan us to improve that, it's certainly welcome. But what this means is the underlying reason that you get that error message is because you have multiple values that would end up in a single cell. Here, just for each combination of female male or the class, I had only one value to put in that location. Of course, with the 900 rows, I have many males that were traveling in the first class and for the other combinations. So pivot only does reordering and cannot do anything with multiple values that would end up in the same cell. So then it raises an error. But of course, it's a typical case that you have multiple values and you want, for example, to take all those values for the single combination, you want to take the mean, for example, the average. So, yeah, here just to show that we have multiple values with the same combination of the sex in the class. But if you need to do that, you can't use pivot, but pivot table. So the pivot table works very similarly as pivot. The only thing, so you also supply what should by index be, what should my columns be, what are the values that I want to put in the resulting frame. But now this will take, by default, the average of all values that end up in the function. And there is also an argument that you can use which is called, this, my aggregation function. Default is mean, but it can also take, for example, I want to know the maximum value paid by one of those combinations or some other statistics. So in this way, you can make a summary table of your full data sets. Other examples you can do, so the maximum as I already did, you can also specify the count just to know how many people passenger with this combination by passing count. For this specific case, to make such kind of cross tabulation, there is also the cross tab method which you can use and which results in the same data set results. But in principle, so this is the cross tab is just a pivot table with a count method to aggregate your data. Okay, some exercises to use this pivot table, yes? There are some, but I don't think I know them. So the question is to get the percentage. You can get margins. I'm not sure, so there is a margins method attribute. And then you get the sum from all rows and all comms, but I'm not sure that you can automatically divide them. Apparently it's not in the doc string anyhow. But in principle, if you want the percentages, you can of course just calculate the numbers and divide it yourself by the total number of passengers. Okay. Okay. Okay. So for the first exercise, I asked for to make a pivot table of the survival rates or survival ratio, I'm sure, but the best term is of the passengers. So what I mean is the fraction of the passengers that actually survived. Since the data, so if you look at the data, the survival column is just zero or one, depending on the person, the passenger died or survived. So to have to know the fraction of people that survived, so I want to count the ones and divide by the total number of rows. In this case, because that are zeros and one and counting the ones and dividing by the total is actually taking a mean of that column. So if I take of the survived column of the full data frame in dot mean, I get zero dot 38, which means that 38 of the passengers on the Titanic survived. If I want to know not on the full data frame, but for different classes or for the separately for males and females, I can do such a pivot table. So I provide my P class, my sex as my index and column values, and I want to calculate the mean of my survived column like that. And here you get a summary table with the percentages, if you would multiply by 100, the percentages of the people that survived the Titanic. I also provide here some code to make a plot of this. And then you see that the females had a much higher chance to survive and also the people in the first class, so the richer people had a higher chance to survive. For this last one, I added, yeah, to get the difference between under and above Athene, so for aged and underaged passengers. Therefore, I just first added the column, and then I can, based on this new column, do this calculation. So this gives me true and false, which is not that nice. Last time somebody proposed, you could also have added a column and that's nice to show in a different way. For example, let's say I just want to split, do some calculations based on different bins of my age column. I could also do, use a cut method of pandas, which does a binning. So I can provide the age column, and then I have to supply, I think, the number of bins. For example, I can say three bins, and this will bin all my data, just in three equal bins. But of course, these are not very nice bins, so another possibility is to supply the actual, so for example, from 0 to Athene, from Athene to, let's say, 15 and then to 100. Like that, and then you get those, for each value in which bin was included. Even now, I can call this, for example, not under age anymore, but age class. And then I could make such a summary table based on the age class. Like that, and then you see your different values for the different bins of age. If I do this for a survived, you will also see that, actually for females, how older you are, the higher chance that you survive, but for the males, the younger you are. Apparently. Okay, we saw a pivot, which was to, or the pivot table, which was to summarize our table, or the original pivot also to go from a nicely column-oriented data where each row is a record, to some kind of summary table. If you want to inverse, we have such a table, we want a summary table, but you want to create a tidy data set where each column is a variable, and in each row is a record, you can use the melt methods. So, I'm first going to calculate such a data set. So, here is now a pivoted frame. So, let's say that I want to convert this to a data frame where I just have columns for the sex, a column for whether it's male or female, and a column with the actual value. So, where each value is one row. Each record is one row. For that, I can use the melt methods. Now, by default, it will put everything under each other, so that's not completely what we want in this case, but it's the key words are not the most user-friendly, but you can say my sex column here, that's already a variable that I want to keep as a column. So, it's only the remaining columns, those columns, that I want to convert to a single column. By supplying here sex like this, I can go from the original one, from sex versus the class. Now, I have a single data frame where you have one value on each row. This one? So, that's the pivot. So, if I have this one, I will call it not DF, but for example, Melted. Melted now looks like this. So, that's what we saw in the first one. I can pivot this, but then I have to specify as my index, I want the, for example, the sex as the column. I want the variable. Yeah, this, due to the melting, it hasn't a nice column name anymore, but you can change it like that. And then the values is my value. Then you get back that one. So, to go from such a tidy data set to this, it's pivot to create the tidy data set, it's Melt. Okay. Melt and pivot, we have seen, but I want to show now a different way to do such reshaping using the stack and unstack method, which are a bit lower level functions, I would say, but also very powerful methods. So, it's not the easiest functions to explain, but basically, if you look at this figure, and it's very similar as to the pivot and the melt, but if you want, you have multiple columns, and we want to put them under each other, you stack the columns. You can use stack. If you have a stack data frame, so with different parts under each other, which you want to put next to each other, you go the other way, then it's unstack. The main thing you need to know with stack and unstack, if you want to use them, they work not on values in the columns, but on your index. So, you need to create, for example, to do the unstack, you need to create a multi-index. But I will just show a small example. So, for example, this data frame, if you want to use, so we have 1, 1, 1, 2, 2 in column A. So, suppose that I want to actually convert this, instead of putting them under each other, I want to put them next to each other. For that, it's going that direction. So, from the right to the left, it's unstack. Before I can use unstack, I have to set them as my index. This creates a multi-index, because I now have two levels in my index. So, you see the results here. Now, the A and B columns are now my index levels. And now I can call unstack on this result. You will see the A and the Bs have now moved to the column labels, and the 1 and 2 are still there. But for the rest, the values just stay the same. So, it's like pivot. It's just reordering, rearranging your data without doing any calculations. So, it's the original data as you had here, the 0 up to 3, but now in a different arrangement. Stack will automatically switch the last index level to the columns, but you could also have specified I want to the first level, so 0, to move the 1 and the 2 to my columns and leave the A and the B in the index. And like this now, you have 1, 2 as your column labels, A and B in the index. So, that's also a way that you can rearrange your data. If you have something like the above, but stack does is putting them in each other, so then you get something like this. So, the 1 and the 2 are now moved back to the index and you get a multi-next. I still have 10 minutes, so this is rather abstract, this stack and unstack, but once you use it, you will see that it's actually a very nice function. So, to use it, I have the following exercise. We saw before a pivot table. So, for example, for the Titanic data set we used, we calculated this pivot table using this function, so it was the survival ratio for the female males versus the class. So, I now want you to try to calculate the exact same table to get exactly the same results, but without using pivot table, but using stack or unstack. As the exercise also says it, you will also have to use group by, because what is actually a pivot table? You have two things on which you want all combinations. So, in this case, it was the sex and the class, and for those groups based on the combination of those two, you calculate the mean, which is typically what you also do for group by. You can use group by a certain key or multiple keys, and for all those groups, you calculate something. So, what you need to do is first use group by to do the actual calculation to get those values, and then you can use unstack in this case to rearrange the data to get the same as there. So, let's try this. Okay. Okay. Okay. Okay. So, first step, what did the group by do? It calculated the mean for the combinations of female males, the sex versus the class. If I want to do such a calculation, I can use group by, but it's grouping by two different variables, so I can supply a list in the group by method. So, I group by, what was it? I group by the class, and I group by the sex like that. Of those groups, I want to calculate the mean. But in this, I forgot, I only want this for the survived column, like that. So, just on the group by object, it's similar as on just a data frame, you can select a single column using this syntax, just inside the square brackets. So, now, I actually already have the correct values, and if you would compare them, those are the same values. So, I have the grouped, calculated the average for those groups. If I now want it in the same arrangement, so the nice table that you get, a table from the class versus the sex, what do you need to do? We want to move this level of my index to the columns, which is an unstack operation. Like this, and now, you will see this is exactly the same table. So, a pivot table, it's a very easy to use function, but it's actually nothing more than a combination of group by and unstack. So, you group by two different things, and then you rearrange the data a little bit. Like that. So, to finish, let's do the same for melt. So, let's say, suppose we have the pivoted data frame, we want to use PD.melt, or we can use melt function to convert it to a tidy data frame. So, now, I have a sex column where male and female are repeated, a column with which class and a column with affairs. So, nicely, one passenger for each row. Last exercise, let's try to do the same. Let's try to, starting from the pivoted, from this data frame, obtain exactly the same as what is in the, what you see there, from melt, but without using melt, but with using stack or unstack. So, the question was why I set the name to non here. So, let's look at that. So, the reason that I did was, now you have here the p class, which looks like a column name, but actually isn't a column name, because it's the name of the index. Because if you look at the actual column names, you see it's only six, one, two, and three. I just, yeah, and by setting the name, sorry, the confusing thing is if it would have been the name of the index, it would be rather useful, but it's actually the name of your columns. So, columns can have, the columns itself can also have a name. And in this case, due to how the pivot table worked, the p class has become the name of the columns. So, you see here, the columns attribute, which is actually an index object, has a name, which is p class. But it's not fully correct anymore, because it's, of course, the p class stands for the one, two, three, but sex is not a class, so therefore I moved that to make the final table, like here, a bit more. Of course, if you have a table like this, you have to know what the one, two, and three stands for. So, there are other ways, but for the exercise, I removed the name. The group by methods. Yeah, for the previous one. Yeah, that's, I would say, so for the group by method, you need to already know a bit more about panels, because you need to know that you can group by two columns, that it creates a multi-index, but of course, you don't really use the multi-index here, but, and then you need to know unstack. So, but once you know those, I personally, just by, yeah, my reflex is also to just use group by and unstack, because there are concepts that I know. So, it's a bit personal preference that you want. It gives exactly the same. So, pivot table is easier to use for if you don't know those other concepts yet, or you know, for example, pivot table from other software where they use the concept of pivot table. So, but that's, yeah, it's just a bit personal preference, which of the two you would like. Now, for the final exercise, what's the first step that I would need to do with my pivoted data frame? Somebody already tried? Yes? What did you, what's the first thing that you did? I don't know, you're all right, but I guess the step is there. Yes, indeed. So, first we need to set the index if we want to use stack. So, because the reason that we need to do this, so what I did was, or what in the solution is, I first set, I will take this one. So, this one. So, I set as the, six as the index. The reason for that is, because if you would just do un-stack, you can look at how it, you just do stack. So, what is stack doing? Putting all columns under each other, and you get it like this. Just all values under each other. But in this case, it's not exactly what you want, because you only want to put the numerical values under each other and keep the female and male and repeat them. So, stack will put all your column names, columns under each other. So, to prevent that sex is also stacked, we just set it as the index. So, it's no longer a column. So, if you look at this one, so now sex is my index, and it will only put the one, two and three under each other. So, I will copy it so we see the different steps. If I do now stack, you will see, and now it are the correct values that are stacked on each other. And you have the female and the male, and the one, two, three are repeated. The female and male are also repeated, but because it's a multi-index, the representation doesn't show them as repeated. So, if we want to get the original frame as we saw here, we want to have them as a data frame with columns. So, we can use reset index to now convert. I will put it in another. Oops. You know the cell, so we see the different steps. So, if you do reset index, it will just, my index levels, there are two index levels, convert them to actual columns. So, then you get something like this. So, here you will see that the melt is actually more convenient as the method using stack. And the main reason is because now you still have the same frame as in the beginning. But, yeah, because it's just the default of the conversion, the column names we have now are not that useful. So, you will have to rename the columns to get the actual exact same result as here. So, the values and the order we already have, so only the column names we don't fully have yet. So, you can, if this is your result, you can do a rename to rename your columns. There are some multiple ways to do this to rename your columns, but one way would be to do a rename of the, you want to rename columns, I think they're getting a bit long, so I will put it on the next line. And you could rename level one to, that's the class, and the zero to the survival or something. Not sure exactly where it was. Oops, and I did something wrong. I need another. So, in this way, you can rename your columns. Just let me check. Yeah, so I did hear something else, something similar, but of course there are other ways that you could rename your columns. So, here you see a way to use stack. In this specific case, I would say the melt is easier than this method because you have to set the index, reset them afterwards, and your column names are a bit mangled. But certainly for some other cases, a stack can also be very useful. Okay, so it's time. So, this was the material that I prepared. The third notebook that I mentioned is mainly an application of all the things that we saw from both the reshaping as the time series functionality, but applied on a single case study on air quality data. So, if you just want to practice a bit more, the concept that we saw, you can always look at this notebook and try to do the exercises which are in there. So, the solutions are always provided, so you can check if you're correct. Okay, that was it. There are any further questions? Yes. So, in this, what you see here, indeed it will return, for most things, it will turn a copy on every step. But whether you do it with chaining those methods or by just each time assigning them and do it step by step, that's the same. You have an in-place keyword that people sometimes use. So, typically, let's say, I'm not sure if Reset actually has this. So, you could then do Resets. So, in principle, you could also, instead of assigning this back to that and then doing the last step with a rename, et cetera. And another way to do the same as I did below with the chaining of the methods, it's exactly the same. You have an in-place method that people sometimes use by doing in-places true. And then you can remove that one. But for most cases, this actually is just syntactic trigger for exactly the same, because many operations will need to do a copy anyhow. For ResetIndex, you don't need to make a copy of the other columns. I'm not sure in this case. But, yeah, in general, the general rule is that many operations in pandas take a copy of your data. But there are, if you have big data, also ways to try to avoid it. Okay.
 So the material is going around, and there you can also download it.  There is a bunch of material in there.  It's a panestero for a full day.  We are going to do the notebooks five and six.  Probably seven we won't have time for, but there is still some additional material on  similar topics as the five and six.  So maybe just to introduce, I'm Joris Van Bosse.  I'm currently working at INRIA in Paris for the Paris Cyclist Center for Data Science.  So this is an advanced tutorial.  Advanced panesthetics is always a bit difficult to choose because there are many and I just  choose some time series data and some reshaping.  Of course, it depends on what you already did with pandas if that will be advanced for  you or not.  So it's maybe for some people it will be too easy for other two difficult.  But there are a lot of exercises in the notebook.  The solutions, you can also find them in the repo.  I will directly show it.  So if I go too slow for you, you can always advance and see the solution yourself.  So that's certainly possible.  For the rest, I like to do hands-on and that you just try it out yourself.  The exercise is just right and I'm here so ask questions.  If there is something not clear, if there is something you don't understand, ask as  many questions as you can.  So we're going to start with the number five time series data.  So if you look in the, if you open the Jupyter notebook inside the directory of the tutorial,  you will see here there is a number five time series data.  So it's that one that you have to open.  So, as you will see, the questions are indicated by green boxes.  So that means that I would like you to try it yourself.  If you tried it, you can always check with the provided solution by uncommenting this  cell and running it.  And then you will see a possible solution.  But I advise you to first try yourself before looking at it.  So the first of the two topics I want to talk about are time series data with pandas.  So it's time series data is something that I find very, yeah, pandas very useful for  and easy for to use.  I will try to show some of the features that are available in pandas when you have time  series data.  So of course, I assume that everybody knows this in the standard library of Python, you  have the daytime objects.  Just as a reminder, you can construct a daytime like that.  You can print it or format it as you want.  In pandas, we have our own custom daytime objects.  We're subclass from the standard library one.  So in principle, everything that you know about the standard daytime object, its attributes,  its methods should work on the pandas object as well, but with some additional features  in pandas.  So the daytime is called timestamp.  You can construct it like this from a string.  As with the daytime object, you can also pause the individual year, month, day, et cetera,  but you can also just construct it from a string.  So again, you can access certain attributes.  For example, I want to get the month of my timestamp.  And apart from that, we also have a time delta object which is similar to the daytime.time  delta in the standard library.  And with that, you can, for example, add a few days.  Now, of course, typically, you won't construct your times yourself by typing the string.  You parse them from somewhere.  And because there are, unfortunately, many different formats that people use, we have  also a quite flexible or powerful method to parse strings that represent day times.  And this method is called two-day time.  You can also read the CSV file and let a column with daytime data be parsed while reading,  but on the root, it will also just use that.  So the two-day time function is the workhorse to parse strings.  Like this, you can parse a clear format, an easy standard format, and then it will parse  it correctly.  Of course, once you start doing other things like the 9th of December like that, you will  see that this is parsed incorrectly if for European.  For somebody from America, it might be correct.  But first, it seems wrong.  So there is one thing you can do.  You can use day first is true to indicate I want to have it parsed assuming that the  day comes first and then only the month.  And that's maybe the main message here is if you have such custom formats, always it's  much better to just specify the exact format on which it will be parsed, like that.  If there's somebody who still needs the material, the stick is there.  And afterwards, there is somebody there as well.  So very basic two-day time we can use for that.  But now once you have your data inside a data frame or a series, we can work with those  data.  For example, I constructed here a series of strings.  I'm not sure what happened here.  I think this comma shouldn't be there.  Apparently, it's OK for two-day time in the comma.  But typically, I don't think that was meant like that.  So we have a series of strings.  Again, like I showed before, with single strings, we can parse it to day time.  And now we have timestamp data.  You see the difference?  The day type here is object.  Object means just any Python object possible.  But in Pandas, this is also used for strings.  Because we don't have a specific data type for strings, we use object.  For the second one, now you see this is day time 64.  So that's a non-py data type to represent day times.  So once we have a series with a correct data type, we get some additional features.  Like we could access on the single timestamp object.  For example, in our or the month attribute as we did before, we can now do that for the  full time series at once.  But they are not directly on the series available, but through.dt attributes.  For example, with.dt,.hour, I get the hour of each entry of my series.  I can get the weekday.  And there are many more.  If you do a time completion, you will see you can also get the day of the year, the hour,  the minute, or you can convert it.  The weekday name, which week of the year, et cetera.  A lot of attributes are available.  If you just want to construct some regular time data, you can also use the date range  function to create some data.  Now some additional features, it's already useful to have it in a column.  But if you have time series data, when you have those data in an index, there are even  more additional features which become available in pandas.  And from now on, we will use a small example, a data set.  It's read it with read CSV.  If I look at it, it has one column, the timestamps, and three columns which are measurements of  this charge in a certain river in flameless.  So it's the flow through the river throughout time.  As I showed before, we want to convert our strings to daytime.  So we use two daytime.  Then I also want to use it as my index, as I just said, so for that I can use.setIndex.  I did it multiple times.  I got an error because I executed that cell twice.  The second time, there was no column time anymore to set as the index.  Therefore, it raised an error.  So now you can also see it in the representation.  Now you see this is my index of my series.  I could also have done it directly when parsing the CSV file by indicating that the first  column should be my index and that I want to parse those as dates.  So if you now look at the index of my data frame, you will see that this is called a  daytime index.  So if you want to have those features in pandas, make sure that it's a daytime index.  And again, just as with a series or a single timestamp, we can access several of those  attributes like the year, day of the year, year itself, et cetera.  So you see it's data from 2009 up to 2013.  So up to now, these are rather the basics of working with date times.  My screen is now a bit too zoomed in, apparently.  So if we plot it, you will already see that pandas, because it's a time series, it has  a daytime index.  The axis of the figure is configured.  And you can also, if you zoom in, you will see that the formatting of the axis automatically  updates to a nice default, as nice looking possible, labels.  So up to now, mostly the basics, as I just said.  But now I want to show a few things.  Once you have...  How to select data if you have a time series.  So in pandas, we can slice our data, slice the rows by providing here, inside the square  brackets, I could do the same actually by using.lock to be more explicit that I want  to slice based on the actual labels of my data frame.  But you can pass the single labels, for example, I want to select my data from 9 to 7 in the  evening on that particular day.  So if you do that, you select those specific days.  But one additional handy feature is that instead of passing the actual timestamp object, which  match our labels, you can also pass it as just a string.  Because it's a daytime index, pandas will try to...  The slides that you provided will automatically...  It knows it has to be dates, so I will try to parse them as dates.  Therefore you don't always have to write the full timestamp like that.  So this is exactly the same.  An additional thing to this, this basic parsing using strings is fine, but there is an additional  feature which is a bit more magical is that if you provide a shorter string, for example,  2013, it's interpreted as a slice from the beginning to the end of 2013.  So for example, if I execute this, you will see it has selected data from the beginning  of 2013 up to the end.  You could also say, for example, I want all data of the first of January.  And then it's...  Again, because it has a higher resolution, it represents multiple rows.  It selects all the different hours of that day.  It's very useful in sometimes an interactive work, but also more magical to use.  So for example...  Yeah, so because it's label-based, it's based on actual labels in my time series, then it's  inclusive.  So if you would slice...  I will show it here.  If you slice up to 18, the 60-clock, the 18 is included.  The 19 was not because there was no 19 there, but so this shows it better that the endpoint  is included.  So that's something basically with Pandas indexing.  You can either index based on actual labels, then the endpoint is included.  So if you say I want from 9 to 6, 6 is included.  You can also slice by position.  I want the first, the second, the third row.  In that case, the NumPy rules for indexing apply, and then the endpoint that your supply  is not included.  Just to quickly show this, if you would do...  So you can use the ILOC1 to do positional indexing.  So here, this will not select the rows 0, 1, 2, but only 0 and 1, and the 2 is not included.  So this gives you two rows instead of including the row at position 2, which is the third  row.  Okay, some small exercises to try this yourself to select some data of the pool frame.  So if you have any problems while trying this out, just raise your hand and ask.  Now, if you theta here means I am sitting here, then I have 5 times a veteran sponsored  standard, which toward those麵s it affects, if you tune out to 1 large F3 you won't  be able to find your direct random strength.  I'm not sure why.  Can you go up?  No, I haven't loaded.  No, no, no.  But if it doesn't, you can't ignore it if you don't have it.  But I suppose you can do it this way with the index directly, right?  So without creating an ulceric code.  Yeah, maybe I showed both of them.  I'm not sure anymore.  Yeah, but you need to...  Oh, no, I didn't see it.  Maybe it's from an older version.  Yeah, you don't have to.  Yeah, so there I need it.  Yeah, but you don't have to.  I mean, you can...  Because it's serious.  Actually, it's a new feature only in the latest.  You see?  Before, it only works on series, and it's only from the 0.20 release.  That also works on the index.  So now you can do it directly.  Index month is in.  But before, it didn't work.  Therefore, there was that solution.  But I suppose that index.month returns a new series.  It returns an index.  No, the thing is, it was already...  Yeah, I remember, because I actually did a PR for that myself.  So the thing is that data.index.month previously returned an empire.  Which doesn't have an in-minute, but it changed to return an index.  Just the same format index as the original index, an integer index, and then you can use it.  So that's the reason that...  That's askew of...  I didn't succeed in installing Jupyter.  Do you have internet?  I have internet, but it seems not to be running very fine.  I had tried to install it with pip, and it broke off after.  And do you know what the error message was?  I have to find it again.  I think it's another terminal.  I have closed it, maybe.  And now did you install Python?  Python works, that is not a problem.  And you have anaconda, but normally it should be included with anaconda, if you try to install anaconda.  There is the problem.  I have a 32-bit system.  I didn't succeed in installing it.  Normally, I think they still have downloads for 32-bits.  I had tried, I think, and it didn't work.  I would try to install the downloaded 32-bits of anaconda.  Then everything is available.  Because everything here seems to be based on...  Yeah.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  The question was, if it's possible to pass a callable to sample,  to re-sample.  But do you mean the callable to calculate the resulting value  based on...  I don't have a answer.  I have a more specific question.  I want to make a result in all of it  by a specific function which will be in the top key.  I think it's probably then easier to use group by  and you can calculate based on our function,  you can calculate the value that you want  and then that's for.  I think it's somehow robust.  The reason is weak, the function is very convenient.  I use it in any production system.  This is something I probably see is rather robust.  Yeah.  The change over time, I don't know if you have a chance.  I think things like group by and resample,  as long as you are using the, not too exotic things  and for example the building methods  on the mean, the median, things like that,  normally that should be stable.  It's more when you maybe supply your own custom function  and there in the, sometimes there are many corner cases  where the function can return and things like that,  that sometimes things can go wrong  or can be fixed or can change.  But certainly for the resample and group by,  working with those time series,  if you use the typical building methods,  normally that should be stable.  You think when they come up with some time,  version 1.0, it should be...  But I show here, so the resample syntax has changed  some reasons, some versions ago.  So that should stay the same as this,  as I show it here.  For group by, resample, rolling operations.  Because resample before was a method where you  gave the how to resample and that has changed.  But now that should be fine and should be stable  for the full panacea.0 series.  Okay, before we go to the next notebook,  quickly go through some of those exercises.  So, yeah, the first one, plot a monthly standard deviation.  So we want monthly, so resample by month.  Take the standard deviation and we plot it.  So here you see a plot of the standard deviation by month.  The next one I want to show, because there is  an additional thing that can be useful,  is I ask for the mean and median value from a single column.  You can, of course, do it like this.  Yeah, I provide a certain selection here,  but I resample and I take the mean,  I resample the same and I take the median.  But what I can also do is if I take this one out,  so this gives me a single column and I resample monthly.  But if I want to calculate multiple things of a single column,  I can also pass a list to aggregates.  And I can say I want the mean and the median, like this,  and to calculate both at the same time  and you get the data frame with different columns  for the different functions, like that.  So if you also want to plot this, oops.  My mouse.  Like that.  Because now you have two columns with a name,  mean and median, you will also get on the plot  on the quick visualization, you also get the legend already there  and you don't have to edit yourself.  Of course, you might want to edit it out of the default books.  So the function that you specify by strain,  like mean and median, they're very fast  because they're optimized at the C level.  So they're on the C.  There was a lot of work done with sci-fi  by being able to pass C callback functions  so that that can bring this down on the river.  Is that also going to be included in the address?  For now, I don't know of any plans through that.  I'm not sure how easy it will be  because the...  Yeah, the site and implementations of mean for goodbye  are very specific for goodbye.  Not generic mean functions, but then applied  at the site and level on the different groups,  but specifically written for goodbye.  So I'm not sure how easy that would be,  but it's mainly that last part  because they are specifically written for goodbye  that they are faster.  So yeah, I'm not sure about it.  Yeah, I will show this one as well.  So yeah, plot the monthly, minimum and maximum daily  average value of that station.  So I first calculate my daily averages  and then based on those data,  I can do another resample and take of those daily values  the monthly, minimum and maximum.  And again, I pass a list to the AC function to do  to calculate two functions at the same time.  And then I get it like that.  So a plot with the minimum and the maximum.  I want to go to this last one.  Maybe it's not available  because I only added this on the...  just before the tutorial.  But an additional exercise which was not included yet.  So calculate the typical yearly pattern  with a monthly resolution.  Because we already saw on the data  if I go look at the plot that I showed,  if I take here the sum for example,  you can already notice each time in the winter,  as can be expected, there is more flow,  a larger flow rate in the rivers.  So here we can calculate monthly values  over the different years.  But maybe I want to make a plot  of how the typical and average yearly profile looks like.  So that's the last exercise.  And the reason that I want to do added here  is because it's also something...  if you work with time series data,  typical to do for example the daily pattern,  yearly pattern.  It's not with resample that we will do it,  but we can do this with goodbye.  So my data frame which was called...  yes, data.  So instead of resampling every month,  which will take every month of every year  a separate values,  I just want to group by all the months  of the different years.  So if I take the month attribute of the index,  this will give the same value  for all the different years for the same month.  So if I want to do something similar as resampling,  which is also for a certain group,  calculates the average or the sum.  But now based on those, I can use group by.  I can first assign this month to a new column  but I can also give just those values to group by.  So now we do a group by the one up to 12  of the different months.  Take the mean for example like that.  So now we get the average for every month,  but over the average over the different years.  If I plot this one, you will see  the typical pattern of the flow rate in the river  in the winter month.  So you have much higher flow rates  as in the summer months.  So depending on what kind of averaging you want to do,  it's resampled or group by that can be very useful.  Okay, are there any remaining questions  about time series data?  If not, I want to go for the last 35 minutes  to the next notebook, which is reshaping data.  To go into some of the functions in Panus  that you can use to reshape your data.  Okay, so reshaping your data is also something  that typically you need to do sometimes with your data  and there are some different ways to approach this in Panus.  I will skip the comparison with Excel for now,  but I will just go through this part.  So pivot to start with this example.  So I made up a small data frame.  It's a part of the Titanic data set about  the different passengers of the Titanic,  whether they survived or not, survived column,  whether sex was in which class they traveled,  the fare they paid to travel with the Titanic.  Now we have columns and for the different variables  and every row is one record.  So one passenger.  You could want to rearrange your data  and to show not a single,  to not have every row as one record,  which is also called your tidy data.  Your columns are variables, your rows are records,  but sometimes you want to rearrange them  to summarize them differently or to present them differently.  To only rearrange your data, you can use the pivot method.  What do you supply in the pivot method?  You say you supply three columns,  the column you want as your new index,  the column you want as your new column labels,  the column for the values to fill in this new data frame.  So if I do this, you will see.  So now I set the class should be my index.  So the one, two, three are now here.  My index labels, the columns labels are my female and male  and those values in this data frame  are the columns with the values from the fair column.  So the pivot method in this way, you can rearrange your data.  Or another example now with the class and the sex,  but the values that I filled in was whether they survived or not.  Now, the data frame that I showed here was only a small subset of the actual data.  So if I read the full data set of the Titanic,  it has not only six values,  but if I look at the info,  so it has almost 900 entries.  So let's suppose that I want to make a similar summary table  with the female male as my columns,  the class as my index rows,  and the fair as my actual values.  So if I try this, I get an exception,  index contains duplicate entries cannot reshape.  It's not the best error message.  I have to say so if somebody wants to do a PR to pan us to improve that,  it's certainly welcome.  But what this means is the underlying reason that you get that error message  is because you have multiple values that would end up in a single cell.  Here, just for each combination of female male or the class,  I had only one value to put in that location.  Of course, with the 900 rows, I have many males  that were traveling in the first class and for the other combinations.  So pivot only does reordering and cannot do anything  with multiple values that would end up in the same cell.  So then it raises an error.  But of course, it's a typical case that you have multiple values  and you want, for example, to take all those values for the single combination,  you want to take the mean, for example, the average.  So, yeah, here just to show that we have multiple values  with the same combination of the sex in the class.  But if you need to do that, you can't use pivot, but pivot table.  So the pivot table works very similarly as pivot.  The only thing, so you also supply what should by index be,  what should my columns be, what are the values that I want to put in the resulting frame.  But now this will take, by default, the average of all values that end up in the function.  And there is also an argument that you can use which is called, this, my aggregation function.  Default is mean, but it can also take, for example, I want to know the maximum value  paid by one of those combinations or some other statistics.  So in this way, you can make a summary table of your full data sets.  Other examples you can do, so the maximum as I already did,  you can also specify the count just to know how many people passenger with this combination  by passing count. For this specific case, to make such kind of cross tabulation,  there is also the cross tab method which you can use and which results in the same data set results.  But in principle, so this is the cross tab is just a pivot table with a count method to aggregate your data.  Okay, some exercises to use this pivot table, yes?  There are some, but I don't think I know them.  So the question is to get the percentage. You can get margins.  I'm not sure, so there is a margins method attribute.  And then you get the sum from all rows and all comms, but I'm not sure that you can automatically divide them.  Apparently it's not in the doc string anyhow.  But in principle, if you want the percentages, you can of course just calculate the numbers and divide it yourself by the total number of passengers.  Okay.  Okay.  Okay.  So for the first exercise, I asked for to make a pivot table of the survival rates or survival ratio, I'm sure, but the best term is of the passengers.  So what I mean is the fraction of the passengers that actually survived.  Since the data, so if you look at the data, the survival column is just zero or one, depending on the person, the passenger died or survived.  So to have to know the fraction of people that survived, so I want to count the ones and divide by the total number of rows.  In this case, because that are zeros and one and counting the ones and dividing by the total is actually taking a mean of that column.  So if I take of the survived column of the full data frame in dot mean, I get zero dot 38, which means that 38 of the passengers on the Titanic survived.  If I want to know not on the full data frame, but for different classes or for the separately for males and females, I can do such a pivot table.  So I provide my P class, my sex as my index and column values, and I want to calculate the mean of my survived column like that.  And here you get a summary table with the percentages, if you would multiply by 100, the percentages of the people that survived the Titanic.  I also provide here some code to make a plot of this.  And then you see that the females had a much higher chance to survive and also the people in the first class, so the richer people had a higher chance to survive.  For this last one, I added, yeah, to get the difference between under and above Athene, so for aged and underaged passengers.  Therefore, I just first added the column, and then I can, based on this new column, do this calculation.  So this gives me true and false, which is not that nice.  Last time somebody proposed, you could also have added a column and that's nice to show in a different way.  For example, let's say I just want to split, do some calculations based on different bins of my age column.  I could also do, use a cut method of pandas, which does a binning.  So I can provide the age column, and then I have to supply, I think, the number of bins.  For example, I can say three bins, and this will bin all my data, just in three equal bins.  But of course, these are not very nice bins, so another possibility is to supply the actual, so for example, from 0 to Athene,  from Athene to, let's say, 15 and then to 100.  Like that, and then you get those, for each value in which bin was included.  Even now, I can call this, for example, not under age anymore, but age class.  And then I could make such a summary table based on the age class.  Like that, and then you see your different values for the different bins of age.  If I do this for a survived, you will also see that, actually for females, how older you are,  the higher chance that you survive, but for the males, the younger you are.  Apparently.  Okay, we saw a pivot, which was to, or the pivot table, which was to summarize our table,  or the original pivot also to go from a nicely column-oriented data where each row is a record,  to some kind of summary table.  If you want to inverse, we have such a table, we want a summary table,  but you want to create a tidy data set where each column is a variable,  and in each row is a record, you can use the melt methods.  So, I'm first going to calculate such a data set.  So, here is now a pivoted frame.  So, let's say that I want to convert this to a data frame where I just have columns for the sex,  a column for whether it's male or female, and a column with the actual value.  So, where each value is one row.  Each record is one row.  For that, I can use the melt methods.  Now, by default, it will put everything under each other,  so that's not completely what we want in this case,  but it's the key words are not the most user-friendly,  but you can say my sex column here,  that's already a variable that I want to keep as a column.  So, it's only the remaining columns, those columns,  that I want to convert to a single column.  By supplying here sex like this,  I can go from the original one, from sex versus the class.  Now, I have a single data frame where you have one value on each row.  This one?  So, that's the pivot.  So, if I have this one, I will call it not DF,  but for example, Melted.  Melted now looks like this.  So, that's what we saw in the first one.  I can pivot this,  but then I have to specify as my index,  I want the, for example, the sex as the column.  I want the variable.  Yeah, this, due to the melting,  it hasn't a nice column name anymore,  but you can change it like that.  And then the values is my value.  Then you get back that one.  So, to go from such a tidy data set to this,  it's pivot to create the tidy data set, it's Melt.  Okay.  Melt and pivot, we have seen,  but I want to show now a different way to do such reshaping  using the stack and unstack method,  which are a bit lower level functions, I would say,  but also very powerful methods.  So, it's not the easiest functions to explain,  but basically, if you look at this figure,  and it's very similar as to the pivot and the melt,  but if you want, you have multiple columns,  and we want to put them under each other,  you stack the columns.  You can use stack.  If you have a stack data frame,  so with different parts under each other,  which you want to put next to each other,  you go the other way, then it's unstack.  The main thing you need to know with stack and unstack,  if you want to use them,  they work not on values in the columns,  but on your index.  So, you need to create, for example, to do the unstack,  you need to create a multi-index.  But I will just show a small example.  So, for example, this data frame, if you want to use,  so we have 1, 1, 1, 2, 2 in column A.  So, suppose that I want to actually convert this,  instead of putting them under each other,  I want to put them next to each other.  For that, it's going that direction.  So, from the right to the left, it's unstack.  Before I can use unstack, I have to set them as my index.  This creates a multi-index,  because I now have two levels in my index.  So, you see the results here.  Now, the A and B columns are now my index levels.  And now I can call unstack on this result.  You will see the A and the Bs  have now moved to the column labels,  and the 1 and 2 are still there.  But for the rest, the values just stay the same.  So, it's like pivot.  It's just reordering, rearranging your data  without doing any calculations.  So, it's the original data as you had here,  the 0 up to 3, but now in a different arrangement.  Stack will automatically switch the last index level  to the columns, but you could also have specified  I want to the first level, so 0,  to move the 1 and the 2 to my columns  and leave the A and the B in the index.  And like this now, you have 1, 2 as your column labels,  A and B in the index.  So, that's also a way that you can rearrange your data.  If you have something like the above,  but stack does is putting them in each other,  so then you get something like this.  So, the 1 and the 2 are now moved back to the index  and you get a multi-next.  I still have 10 minutes, so this is rather abstract,  this stack and unstack, but once you use it,  you will see that it's actually a very nice function.  So, to use it, I have the following exercise.  We saw before a pivot table.  So, for example, for the Titanic data set we used,  we calculated this pivot table using this function,  so it was the survival ratio for the female males versus the class.  So, I now want you to try to calculate the exact same table  to get exactly the same results,  but without using pivot table, but using stack or unstack.  As the exercise also says it, you will also have to use group by,  because what is actually a pivot table?  You have two things on which you want all combinations.  So, in this case, it was the sex and the class,  and for those groups based on the combination of those two,  you calculate the mean, which is typically what you also do for group by.  You can use group by a certain key or multiple keys,  and for all those groups, you calculate something.  So, what you need to do is first use group by to do the actual calculation  to get those values, and then you can use unstack in this case  to rearrange the data to get the same as there.  So, let's try this.  Okay.  Okay.  Okay.  Okay.  So, first step, what did the group by do?  It calculated the mean for the combinations of female males,  the sex versus the class.  If I want to do such a calculation, I can use group by,  but it's grouping by two different variables,  so I can supply a list in the group by method.  So, I group by, what was it?  I group by the class, and I group by the sex like that.  Of those groups, I want to calculate the mean.  But in this, I forgot, I only want this for the survived column, like that.  So, just on the group by object, it's similar as on just a data frame,  you can select a single column using this syntax, just inside the square brackets.  So, now, I actually already have the correct values,  and if you would compare them, those are the same values.  So, I have the grouped, calculated the average for those groups.  If I now want it in the same arrangement, so the nice table that you get,  a table from the class versus the sex, what do you need to do?  We want to move this level of my index to the columns,  which is an unstack operation.  Like this, and now, you will see this is exactly the same table.  So, a pivot table, it's a very easy to use function,  but it's actually nothing more than a combination of group by and unstack.  So, you group by two different things, and then you rearrange the data a little bit.  Like that.  So, to finish, let's do the same for melt.  So, let's say, suppose we have the pivoted data frame,  we want to use PD.melt, or we can use melt function to convert it to a tidy data frame.  So, now, I have a sex column where male and female are repeated,  a column with which class and a column with affairs.  So, nicely, one passenger for each row.  Last exercise, let's try to do the same.  Let's try to, starting from the pivoted, from this data frame,  obtain exactly the same as what is in the, what you see there, from melt,  but without using melt, but with using stack or unstack.  So, the question was why I set the name to non here.  So, let's look at that.  So, the reason that I did was, now you have here the p class,  which looks like a column name, but actually isn't a column name,  because it's the name of the index.  Because if you look at the actual column names, you see it's only six, one, two, and three.  I just, yeah, and by setting the name, sorry,  the confusing thing is if it would have been the name of the index,  it would be rather useful, but it's actually the name of your columns.  So, columns can have, the columns itself can also have a name.  And in this case, due to how the pivot table worked,  the p class has become the name of the columns.  So, you see here, the columns attribute, which is actually an index object,  has a name, which is p class.  But it's not fully correct anymore, because it's, of course, the p class stands for the one, two, three,  but sex is not a class, so therefore I moved that to make the final table, like here,  a bit more.  Of course, if you have a table like this, you have to know what the one, two, and three stands for.  So, there are other ways, but for the exercise, I removed the name.  The group by methods.  Yeah, for the previous one.  Yeah, that's, I would say, so for the group by method,  you need to already know a bit more about panels, because you need to know that you can group by two columns,  that it creates a multi-index, but of course, you don't really use the multi-index here,  but, and then you need to know unstack.  So, but once you know those, I personally, just by,  yeah, my reflex is also to just use group by and unstack, because there are concepts that I know.  So, it's a bit personal preference that you want.  It gives exactly the same.  So, pivot table is easier to use for if you don't know those other concepts yet,  or you know, for example, pivot table from other software where they use the concept of pivot table.  So, but that's, yeah, it's just a bit personal preference, which of the two you would like.  Now, for the final exercise, what's the first step that I would need to do with my pivoted data frame?  Somebody already tried?  Yes?  What did you, what's the first thing that you did?  I don't know, you're all right, but I guess the step is there.  Yes, indeed.  So, first we need to set the index if we want to use stack.  So, because the reason that we need to do this, so what I did was,  or what in the solution is, I first set, I will take this one.  So, this one.  So, I set as the, six as the index.  The reason for that is, because if you would just do un-stack, you can look at how it, you just do stack.  So, what is stack doing?  Putting all columns under each other, and you get it like this.  Just all values under each other.  But in this case, it's not exactly what you want, because you only want to put the numerical values under each other  and keep the female and male and repeat them.  So, stack will put all your column names, columns under each other.  So, to prevent that sex is also stacked, we just set it as the index.  So, it's no longer a column.  So, if you look at this one, so now sex is my index, and it will only put the one, two and three under each other.  So, I will copy it so we see the different steps.  If I do now stack, you will see, and now it are the correct values that are stacked on each other.  And you have the female and the male, and the one, two, three are repeated.  The female and male are also repeated, but because it's a multi-index, the representation doesn't show them as repeated.  So, if we want to get the original frame as we saw here, we want to have them as a data frame with columns.  So, we can use reset index to now convert.  I will put it in another.  Oops.  You know the cell, so we see the different steps.  So, if you do reset index, it will just, my index levels, there are two index levels, convert them to actual columns.  So, then you get something like this.  So, here you will see that the melt is actually more convenient as the method using stack.  And the main reason is because now you still have the same frame as in the beginning.  But, yeah, because it's just the default of the conversion, the column names we have now are not that useful.  So, you will have to rename the columns to get the actual exact same result as here.  So, the values and the order we already have, so only the column names we don't fully have yet.  So, you can, if this is your result, you can do a rename to rename your columns.  There are some multiple ways to do this to rename your columns, but one way would be to do a rename of the,  you want to rename columns, I think they're getting a bit long, so I will put it on the next line.  And you could rename level one to, that's the class, and the zero to the survival or something.  Not sure exactly where it was.  Oops, and I did something wrong.  I need another.  So, in this way, you can rename your columns.  Just let me check.  Yeah, so I did hear something else, something similar, but of course there are other ways that you could rename your columns.  So, here you see a way to use stack.  In this specific case, I would say the melt is easier than this method because you have to set the index, reset them afterwards, and your column names are a bit mangled.  But certainly for some other cases, a stack can also be very useful.  Okay, so it's time.  So, this was the material that I prepared.  The third notebook that I mentioned is mainly an application of all the things that we saw from both the reshaping as the time series functionality, but applied on a single case study on air quality data.  So, if you just want to practice a bit more, the concept that we saw, you can always look at this notebook and try to do the exercises which are in there.  So, the solutions are always provided, so you can check if you're correct.  Okay, that was it.  There are any further questions?  Yes.  So, in this, what you see here, indeed it will return, for most things, it will turn a copy on every step.  But whether you do it with chaining those methods or by just each time assigning them and do it step by step, that's the same.  You have an in-place keyword that people sometimes use.  So, typically, let's say, I'm not sure if Reset actually has this.  So, you could then do Resets.  So, in principle, you could also, instead of assigning this back to that and then doing the last step with a rename, et cetera.  And another way to do the same as I did below with the chaining of the methods, it's exactly the same.  You have an in-place method that people sometimes use by doing in-places true.  And then you can remove that one.  But for most cases, this actually is just syntactic trigger for exactly the same, because many operations will need to do a copy anyhow.  For ResetIndex, you don't need to make a copy of the other columns.  I'm not sure in this case.  But, yeah, in general, the general rule is that many operations in pandas take a copy of your data.  But there are, if you have big data, also ways to try to avoid it.  Okay.
EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_0_131.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_132_2927.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_2928_3767.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_3768_4199.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_4200_4247.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_4248_4811.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_4812_7619.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_7620_7823.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_7824_8303.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_8304_9527.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_9528_10499.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_10500_10679.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_10680_11387.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_11388_11459.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_11460_12527.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_12528_12587.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_12588_15383.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_15384_15587.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_15588_16259.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_16260_16799.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_16800_17699.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_17700_17963.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_17964_18527.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_18528_18623.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_18624_18827.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_18828_18875.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_18876_19115.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_19116_19379.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_19380_20819.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_20820_21263.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_21264_21527.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_21528_22223.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_22224_22415.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_22416_23435.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_23436_23483.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_23484_25487.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_25488_30287.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_30288_34727.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_34728_34811.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_34812_35987.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_35988_36407.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_36408_36479.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_36480_36527.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_36528_41027.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_41028_41483.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_41484_42263.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_42264_42587.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_42588_42671.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_42672_46871.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_46872_47471.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_47472_47591.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_47592_47939.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_47940_50603.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_50604_51587.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_51588_55211.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_55212_58091.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_58092_59915.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_59916_60011.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_60012_66119.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_66120_66275.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_66276_68087.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_68088_68471.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_68472_69119.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_69120_70727.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_70728_70787.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_70788_71831.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_71832_71891.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_71892_74339.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_74340_74519.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_74520_75539.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_75540_75851.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_75852_76931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_76932_77387.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_77388_77819.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_77820_81167.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_81168_81551.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_81552_81875.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_81876_82223.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_82224_82631.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_82632_82883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_82884_82931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_82932_83339.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_83340_87863.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_87864_88139.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_88140_89519.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_89520_90983.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_90984_91883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_91884_91931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_91932_97391.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_97392_97883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_97884_97931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_97932_99383.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_99384_101615.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_101616_103055.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_103056_103883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_103884_103931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_103932_104327.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_104328_106811.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_106812_107531.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_107532_108131.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_108132_109883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_109884_112883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_112884_112931.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_112932_114683.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_114684_114875.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_114876_117095.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_117096_118319.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_118320_120503.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_120504_121103.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_121104_121175.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_121176_123503.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_123504_124883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_124884_125219.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_125220_126395.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_126396_126779.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_126780_130811.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_130812_130883.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_130884_133091.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_133092_133787.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_133788_135899.png EuroSciPy_2017__Advanced_Pandas_gS7kVg-4ZaU_135900_138660.png