Data is not flat
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Dammer, Alisa
Feature engineering and model training often comes hand in hand. Some tasks have an overwhelming amount of high dimensional data, some tasks have little data or very low-dimension data. This talk targets the latter problem: what can be done with the data itself to significantly improve the model performance and when manual feature engineering does make sense. A sample case of Classification problem with NN will be presented The goal of the talk is to remind about something every person working with the data thinks and probably uses.
 So, hi everybody. Today I will not only talk about, or I will talk about specific topic that is rather not that official in the data science called feature engineering because there is like no discipline that you can learn in the university or find a course on. This is actually considered to be something that you should be able to do, but it's not always that simple. So, and in the end of the presentation you will get by, what I mean by saying data is not flat. So the structure of this presentation will be as following. I will just briefly mention what feature engineering is, what is, it is not, and I will concentrate on a certain example of feature engineering. There are like several types. Then I will give you a classification example that's a really stupid project of mine and you probably will like it at least I hope. So at the conclusion I will again tell you about how to use feature engineering, what you can do almost automatically and what actually helps and works in most cases. And afterwards I hope we will still have time for QA session. So, what is a feature engineering? According to the Wikipedia it's pretty accurate definition. This is a process of using domain knowledge of the data to create features that make machine learning algorithms work. What does it mean? You probably all heard about concept garbage in, garbage out, meaning if you have a garbage data that doesn't even reflect the situation you're trying to work on, you can try whatever model you can come up with, it still won't produce any good results. So feature engineering is not of preparing data the way that any algorithm can perform better. So the feature engineering as it was seen in the description, it is an expert domain knowledge by that I mean that someone who is driving a taxi knows probably a bit more about the whole industry and how the things work rather than some college student. And these process can be both automated and manual. I won't talk about automated process, this is more of a machine learning approaches that cover the feature learning. Like it's some part of a knowledge transfer for instance. Or there are a lot of also Python tools that if you have multiple data like real lot of tables you can concatenate them some way and just run automatic functions to create new features as some sort of transformation from the older columns. I won't talk about that. I will talk explicitly about manual creation. However, all of those approaches are quite expensive. You need time to identify what you're working with. You need time to identify what features might be useful. And sometimes it is important to first take a look at your model. Sometimes it doesn't mean anything at all. And as garbage in garbage outstates, data quality is probably way more important than the actual model. Fitch engineering is not part of the data engineering process. Meaning data engineers are people who collect the data initially, make some initial quick cleaning things like taking completely insane samples out, somewhat fluttering it out if necessary, they make the data available. They have no idea actually what for this data will be used and thus they are not responsible for certain specific knowledge search. And it is not the part of preprocessing, even if it's done by data scientists. Again, these data science is an iterative process. You create a model, you feed some data in if you're not even known how your model will behave, you just watch what happens and then you started tweaking some parameters on the data side on the model side. And this iterative will repeat itself many times. But future engineering comes in question if your model does not perform well at all and even some additional models do not perform well. And you have a kind of sense that the data is at fault. So I said earlier that there are pretty much two main approaches, manual and automated. For automated, like tons of articles, just type in feature selection, feature engineering automatic or feature creation, Python, and you will get all possible blog posts, GitHub repositories, just play around with it. It also pretty powerful, but the thing with those approaches, you need to have quite a lot of data or you need to have high dimensional data. In this case, they will work the best. Basically you will try to reduce the amount of noise or unusual information to feed into your model. Second approach is used when you don't have enough data. We are talking about either very low amount of samples or very low dimensionality of the data or both. So the thing that I want to show you today will be concentrated on the case where I don't have data, not at all, but like thousand samples, it's nothing. And at the same time, it's like one dimension. So what can you do with this? Again I quite like the article on Wikipedia about the feature engineering. It has quite a lot of resources and links to further usage and further sources to read. And they mention as process how to actually approach the feature engineering. So brainstorming of testing features, deciding what features to create, creating features, and checking how the features work with the model, improving if needed, improving meaning you probably have to scale it down or if it's a categorical feature you need to define categories the way that the model actually will understand them and won't have this skewed distribution. So you need to really know your data and repeat it and repeat it and repeat it. So this is the process that we're going to all do today. For example, I said it's a free time project of mine, I have a friend, he is, I don't know, he's a student, informatic student and he has very weird sleeping cycles. You know those typical informatic people who are going to bed at like 7 in best case AM and wake up at 8. And actually he's a quiet big part of our friendship community and everybody is always interested when is he awake. And on Telegram everybody is asking, is he awake? Is he awake? And as a part of the meme or gag you can say, I was given a dare challenge to write a model that will actually try to learn his sleeping patterns and instead of actually asking and waiting for a response, we will get the approximate answer right away. For that reason, again it was a dare project so I wrote yet another wrapper around fully connected neural networks. I did it because I needed it to be configurable my way. It's nothing fancy, it's just NumPy with some features. So about these, an engenerator, it is available on PyPy if you want to play around with it, the feature that I needed was that the whole model should be configurable with JSON so that I will be able to run several different, slightly different models simultaneously or even have pure understanding, okay, where I came from and where I am now. So you can just give architecture, it can be flat. These currently there is only the sigmoid meaning binary classification implemented, this is the last layer and then you can have as many layers as you want to with the amount of nodes within, you have activations for each layer and something like confidence meaning you have the probability of something like the, you don't get the zero or one right away, you get the probability and you need to interpret it. So you can set it to whatever you want, you can have different learning rates, all the hyperparameters pretty much. So originally I was given just thousand samples, these were timestamps and the state whether the guy was awake or not with a certain minutes step. So this is what we are going to use, pretty much we are going to create infrastructure approachable for the building the model, we are going to create and train the model and just see what the prediction and the accuracy is in comparison to the testing set. I don't have a validation set because it is just thousand samples and for the purpose of this presentation it is not needed. So originally I thought what will happen if I just throw pretty much a time series inside and see what happens? Well not that good. You can see that the model performed at 30% accuracy which is way worse than even a coin flip and I can wait just to see what the guy is awake or not. And I thought maybe it is not really a prediction problem, it is not a regression, it might be a classification problem and in this case I obviously lack data but the good thing I have time data and time has certain attributes to it, you can describe certain point on the moment by what date is it to date, not even the date by what day of the week it is and for us people it is important because on the Monday to Friday we tend to work, meaning we have certain behavioral pattern on the weekends, our behavioral pattern might end, might not end, but change and in this case this difference is important. Also it is important whether it is late in the evening, early in the morning or it is lunch time so you can see where I am going there. And I tried to unwrap the timestamp into just vector containing this feature describing each timestamp. So basically I went through brainstorming or testing features, deciding what features to create. This is the day and the hour and I created them and I also checked them how they work. By the way I am using two-layered fully connected neural network, this is approved to be the final best model because after I did some feature engineering I also played with different architectures and this one had the best performance so I take it from the very beginning. So we run this one with two features which are date of the week and the hour. The model suddenly jumps to 81% of the accuracy which is like really not too shabby and that made me thinking what do I else know about this guy? I know that he is a student and he is a human. I know that he also lives in Hamburg where I come from and as any living being in Hamburg we are affected by the weather which is better, tend to be a little bit better in summer and it is completely disastrous in late winter, early spring. So the season might also be important there. And in this case I say okay, let's throw some season into it as a third attribute and now we have as input date of the three-dimensional vector. Now the model performed 8% better than the original or the previous one. The model is the same, the date is just unwrapped. So 89% is already way better than I started originally with but at the same time 95% is something that should be considered to be production ready like 95 and above so I want to go further, I want to push the limits. So what do I know about this guy again? He is a student, isn't important? Model actually yes because he has exam phase, he has some lecture phase and he has vacation. And I know myself and I tend to behave completely different on exams and during vacation. During vacation I slept like whole day through, whole night through. So why should be different for him? And in this case this is really artificial feature because it is static information taken from the website of our university where I and he are students but you probably cannot automate such information collection if you know how or if you trust the sources that can provide you with this information. Anyhow this is more or less static and now we have the four features model which performed to 95% accuracy. For me it was like 95, industry ready, I am done with it. But if I take a step back and look at it, all I did is I tried to reconstruct behavioral attributes or time attributes of the behavior of this guy and I started originally with the only time stamp and I ended up with pretty good results only by feature engineering. You should just believe me, you can obviously try it out yourself but tweaking the model parameters, taking different models didn't perform well at all. I never got better than 45% accuracy on only time stamps. I tried STMs, didn't perform. I tried some models from classic econometrics like Arima but this is not exactly the problem I was approaching. So let's check out whether this guy is awake now. No he is not. It's like three, yeah he will be probably awake in like two or three hours. Ping me on the telegram or whatever just to ask whether he is really awake if you're interested. So this is pretty much what I just wanted to show you. This is very simple yet very powerful and I've only used my, like I am an expert, I know this guy, I've pretty much simulated my train of thoughts, what I know about the guy, what I should, what information do I use and as a programmer I decided I tried to implement or collect this data and yeah, that's everything based only on time stamp. Anything if you have more data that you can unwrap. Another case I can think of is for instance there is a competition on Kegel about taxi. You have to compute the waiting time in the taxi based on the four or three years of taxi time from Mexico or Ecuador countries and there are like normal data you can see in the file they provide. It's geocoordinates of pickup drop off, timestamp of pickup on drop off, the distance they actually capped drove and how long did it took, how long did the whole ride lasted and stuff like that. But you can throw something like, I don't know whether the time, like for instance you can, again can automatically unwrap the time stamps into date, sorry date, a day and an hour and based on day and hour you can for instance decide whether it's a rush hour and you know that in rush hour cars tend to have longer drive and waiting time than normally because the roads are full. Also you can define something like holidays, national holidays. This is where really expert domain knowledge comes in and this is purely manual. I don't really imagine anything comes close to this train of thought and performance by using automatic approach because the data is really not there, at least it looks like it. So it is costly. It took me about three days to, like one day to build a model and one day I thought about okay what can I do? I only have this much data and only this data. What should I do? What helped me originally I've tried to, again reconstruct my own thoughts and I've tried pretty much the model where I choose the fully connected neural networks is also reconstruction of my brain work. For instance for me it was important that today the feature importance is also took some place and the layers they are, they're reflecting my for instance personal attributes that I value more. For instance I know that exams for this guy are not that important but what hour it is is way important and you could see that after adding just day and hour we got 50% of accuracy and by adding only season we got like 5 or 7%. It is very error prone because it's expert domain knowledge you mostly especially in literacy you will have to drag someone in who knows their stuff for instance some engineers or even taxi drivers and between this communication over time between the expert and the programmer, data engineer or data science some information will be lost. Also again it is not really clear at what scale and what features should be incorporated into the model or into the data set. Sometimes you are so happy about the results about the progress that you do that you try to throw in some features that are based on one each other meaning you will land into a multi-colinearity situation where you have quite a lot of noise but not that much meaningful data. So sanity check is always a good thing just to create several features, think how they perform, try to maybe fine tune the model, try to get the best out of this small scenario this local optimal and then think again because every time you need to create a brainstorming session you will be affected by the previous brainstorming session and at some point you will run out of ideas. So feature engineering is not a knowledge transfer it can be used as a part of it basically you train models on certain amount of data that has at the first glance nothing to do with your task but you have to train the first model somehow so it can be useful for that step as well. By unwrapping features you create even more features obviously and you can use feature selection you can use automatic feature creation and what's ever because you have way more space to do that. I think it's very powerful. You can argue with that but I think it's powerful. And it worked with all models. I tried some neural networks, I tried classical approaches like decision tree and they all performed way better when they got more data. It's like yeah, duh, but every time you see it in the code and the results it's wow. So this is pretty much everything I wanted to say about the feature engineering. Thank you for your attention. So thanks for the great talk. There's time for questions. Hi, thanks for the talk. I was just wondering have you at all compared automated feature selection to your insightful feature selection and whether they perform better or worse? Feature selection, like I won't use feature engineering when I have enough data to actually use feature selection. Feature selection comes in first when you have to, the data to search like to take some features out to prepare for training. Feature engineering is pretty much enable feature selection. So I won't compare them on this level. Thank you for the talk. How did you select the frequency about your data, for example? I was given it. I was pretty much just shoved a file, a text file and I knew nothing more than this. So in your case you want to know at what hour your friend is awake, not in what minute or second or so? It will depend on the task, obviously. In this case it was enough. Like certain minutes for humans to know whether somebody is awake or not, available or not, or the real-time application would be an automated system for remote teams that will allocate slots for meetings or some discussions based on their working patterns, where people tend to be available, where you can use, for instance, multi-class models where you put something like I want to do my correspondence, I am available for the talk, I am focusing on new features and so on and so forth. So in the real-time or real-world example you will probably need a different frequency of the data. In my case it was more than enough. Again, based on the frequency of the data, different features will be more or less important. For instance, exams, I was just lucky that I had two university phases in the data sample, otherwise it won't make any difference. Some more questions? Hi, thanks for the talk. You seem to be just advocating adding more and more features until you have improved your model enough. Is there some point that will start to become counterproductive and you will hit the curse dimensionality by adding more and more dimensions to your model? In my case it's rather hard because I honestly, for me, was already kind of, what can I also add? But if you start not with one dimension but more, basically, yes, you can hit the problem where you're trying to get all the data simulating the whole real world, which is the point where you start noticing some patterns or behavior, which we are talking about classification, that is not typical for your case but is more general. For instance, you know there are historical 100 cycles or whatever and there are not only the amount of data important but things that show exactly when it happens, why it happens, so you can learn from features. This is where it starts to be counterproductive. This is why I said define not more than four features. First test them out, try to replace them. It is a pretty small amount of work for the start of four features unless you have real giant model but in this case you probably already not doing something wrong than you started with the feature engineering and such a complex model. After a while, I personally tend to say that depending on the case, how complex is something you want to learn, if you want to learn just the sleeping cycles of a dude, then you don't need much data. If you're trying to build a self-driving car, oh boy, you're going to need quite a lot of sensors, quite a lot of data, quite a lot of features. So basically, depending on the complexity of the task, it depends the complexity of the data and directly amount of features needed. You can estimate yourself how complex your data should be or could be. Reduce this by two and then start working. Okay, any more questions? Come on, I'm not biting. Then maybe I have a question. So I'm not a data scientist so I don't have much clue about this. But I've heard that as you add more information to your model, there's a risk of overfitting. Did you do some sort of cross-validation? Because your precision goes up. But that's sort of too expected anyway, isn't it? Yeah, true. In this case, I didn't because I had way too less data and I was in the territory of always underfitting. The amount of features was totally under control. If I would start to throw something like, does this guy live with the parents? Does he has a pet? And so on and so forth. And probably I will overfit at some point. So basically, in order to not overfit or to have low risk of overfitting, you need to not only keep your data, you need to keep your data still low dimensional. To not overcomplicate the life or description of the situation. Okay. One last question. There's still time, but... What if we come up with a feature that is really not helping with the accuracy then we'll just drop or how can we observe that? There was a thing. When I initially started to creating features, I also thought about, I didn't thought at all, at the first, about the scale. But this was okay because I had like four categories. But at some point I tried to scale them. For instance, you saw here this three features, two three features falls. This is basically scaled and not like row version from zero to whatever. And this is more of a normalized one from minus two to two or so. And this models actually perform not way better. Not always. But I tried to define season like minus two, minus one, zero, one. And it decreases the performance by seven percent. Also some features, as I said earlier, are just a bit different form of already existing. So they won't really decrease your performance. They might have, but they won't help either. Also always check your data whether this, the feature you want to create will be present. I had another case where I was really interested whether some national holidays are affecting taxi waiting time. I don't know, I spent not that much time, like four hours, by really finding out all the information and turns out they just don't have those sample points. So time wasted. Okay, one last question. Thanks. So when you're dealing with these small data sets, low dimensional data, how do you differentiate between the cases when there just isn't enough information to build a useful model and when you just haven't found the right manly engineered features yet? Mostly gut feeling and common sense, if honestly. My differ, again in this case, it was pretty obvious. I simulated my own thoughts about the problem and I figured out what I personally call feature, what my brain defined as a feature. And this is pretty much where I stop. In more complex situations, I probably would start with model first and not the data if I have at least like four features. And after a while, I will notice that no model tuning actually works. And but the data is clean. I have, I don't have missing values. It is sane. And then I will ask what my model wants to describe. And as a human, I'm smarter than the model because I knew more. I've seen more. And this is where I will start trying to dig deeper and bring some features in that model might find sufficient, might not. Thank you very much. Okay. So give a big hand to Alicia again.
 So, hi everybody.  Today I will not only talk about, or I will talk about specific topic that is rather not  that official in the data science called feature engineering because there is like no discipline  that you can learn in the university or find a course on.  This is actually considered to be something that you should be able to do, but it's not  always that simple.  So, and in the end of the presentation you will get by, what I mean by saying data is  not flat.  So the structure of this presentation will be as following.  I will just briefly mention what feature engineering is, what is, it is not, and I will concentrate  on a certain example of feature engineering.  There are like several types.  Then I will give you a classification example that's a really stupid project of mine and  you probably will like it at least I hope.  So at the conclusion I will again tell you about how to use feature engineering, what  you can do almost automatically and what actually helps and works in most cases.  And afterwards I hope we will still have time for QA session.  So, what is a feature engineering?  According to the Wikipedia it's pretty accurate definition.  This is a process of using domain knowledge of the data to create features that make machine  learning algorithms work.  What does it mean?  You probably all heard about concept garbage in, garbage out, meaning if you have a garbage  data that doesn't even reflect the situation you're trying to work on, you can try whatever  model you can come up with, it still won't produce any good results.  So feature engineering is not of preparing data the way that any algorithm can perform  better.  So the feature engineering as it was seen in the description, it is an expert domain  knowledge by that I mean that someone who is driving a taxi knows probably a bit more  about the whole industry and how the things work rather than some college student.  And these process can be both automated and manual.  I won't talk about automated process, this is more of a machine learning approaches that  cover the feature learning.  Like it's some part of a knowledge transfer for instance.  Or there are a lot of also Python tools that if you have multiple data like real lot of  tables you can concatenate them some way and just run automatic functions to create new  features as some sort of transformation from the older columns.  I won't talk about that.  I will talk explicitly about manual creation.  However, all of those approaches are quite expensive.  You need time to identify what you're working with.  You need time to identify what features might be useful.  And sometimes it is important to first take a look at your model.  Sometimes it doesn't mean anything at all.  And as garbage in garbage outstates, data quality is probably way more important than  the actual model.  Fitch engineering is not part of the data engineering process.  Meaning data engineers are people who collect the data initially, make some initial quick  cleaning things like taking completely insane samples out, somewhat fluttering it out if  necessary, they make the data available.  They have no idea actually what for this data will be used and thus they are not responsible  for certain specific knowledge search.  And it is not the part of preprocessing, even if it's done by data scientists.  Again, these data science is an iterative process.  You create a model, you feed some data in if you're not even known how your model will  behave, you just watch what happens and then you started tweaking some parameters on the  data side on the model side.  And this iterative will repeat itself many times.  But future engineering comes in question if your model does not perform well at all and  even some additional models do not perform well.  And you have a kind of sense that the data is at fault.  So I said earlier that there are pretty much two main approaches, manual and automated.  For automated, like tons of articles, just type in feature selection, feature engineering  automatic or feature creation, Python, and you will get all possible blog posts, GitHub  repositories, just play around with it.  It also pretty powerful, but the thing with those approaches, you need to have quite a  lot of data or you need to have high dimensional data.  In this case, they will work the best.  Basically you will try to reduce the amount of noise or unusual information to feed into  your model.  Second approach is used when you don't have enough data.  We are talking about either very low amount of samples or very low dimensionality of the  data or both.  So the thing that I want to show you today will be concentrated on the case where I don't  have data, not at all, but like thousand samples, it's nothing.  And at the same time, it's like one dimension.  So what can you do with this?  Again I quite like the article on Wikipedia about the feature engineering.  It has quite a lot of resources and links to further usage and further sources to read.  And they mention as process how to actually approach the feature engineering.  So brainstorming of testing features, deciding what features to create, creating features,  and checking how the features work with the model, improving if needed, improving meaning  you probably have to scale it down or if it's a categorical feature you need to define categories  the way that the model actually will understand them and won't have this skewed distribution.  So you need to really know your data and repeat it and repeat it and repeat it.  So this is the process that we're going to all do today.  For example, I said it's a free time project of mine, I have a friend, he is, I don't know,  he's a student, informatic student and he has very weird sleeping cycles.  You know those typical informatic people who are going to bed at like 7 in best case AM  and wake up at 8.  And actually he's a quiet big part of our friendship community and everybody is always  interested when is he awake.  And on Telegram everybody is asking, is he awake?  Is he awake?  And as a part of the meme or gag you can say, I was given a dare challenge to write a model  that will actually try to learn his sleeping patterns and instead of actually asking and  waiting for a response, we will get the approximate answer right away.  For that reason, again it was a dare project so I wrote yet another wrapper around fully  connected neural networks.  I did it because I needed it to be configurable my way.  It's nothing fancy, it's just NumPy with some features.  So about these, an engenerator, it is available on PyPy if you want to play around with it,  the feature that I needed was that the whole model should be configurable with JSON so  that I will be able to run several different, slightly different models simultaneously or  even have pure understanding, okay, where I came from and where I am now.  So you can just give architecture, it can be flat.  These currently there is only the sigmoid meaning binary classification implemented,  this is the last layer and then you can have as many layers as you want to with the amount  of nodes within, you have activations for each layer and something like confidence meaning  you have the probability of something like the, you don't get the zero or one right away,  you get the probability and you need to interpret it.  So you can set it to whatever you want, you can have different learning rates, all the  hyperparameters pretty much.  So originally I was given just thousand samples, these were timestamps and the state whether  the guy was awake or not with a certain minutes step.  So this is what we are going to use, pretty much we are going to create infrastructure  approachable for the building the model, we are going to create and train the model and  just see what the prediction and the accuracy is in comparison to the testing set.  I don't have a validation set because it is just thousand samples and for the purpose  of this presentation it is not needed.  So originally I thought what will happen if I just throw pretty much a time series inside  and see what happens?  Well not that good.  You can see that the model performed at 30% accuracy which is way worse than even a coin  flip and I can wait just to see what the guy is awake or not.  And I thought maybe it is not really a prediction problem, it is not a regression, it might  be a classification problem and in this case I obviously lack data but the good thing I  have time data and time has certain attributes to it, you can describe certain point on the  moment by what date is it to date, not even the date by what day of the week it is and  for us people it is important because on the Monday to Friday we tend to work, meaning  we have certain behavioral pattern on the weekends, our behavioral pattern might end,  might not end, but change and in this case this difference is important.  Also it is important whether it is late in the evening, early in the morning or it is  lunch time so you can see where I am going there.  And I tried to unwrap the timestamp into just vector containing this feature describing  each timestamp.  So basically I went through brainstorming or testing features, deciding what features  to create.  This is the day and the hour and I created them and I also checked them how they work.  By the way I am using two-layered fully connected neural network, this is approved to be the  final best model because after I did some feature engineering I also played with different  architectures and this one had the best performance so I take it from the very beginning.  So we run this one with two features which are date of the week and the hour.  The model suddenly jumps to 81% of the accuracy which is like really not too shabby and that  made me thinking what do I else know about this guy?  I know that he is a student and he is a human.  I know that he also lives in Hamburg where I come from and as any living being in Hamburg  we are affected by the weather which is better, tend to be a little bit better in summer and  it is completely disastrous in late winter, early spring.  So the season might also be important there.  And in this case I say okay, let's throw some season into it as a third attribute and now  we have as input date of the three-dimensional vector.  Now the model performed 8% better than the original or the previous one.  The model is the same, the date is just unwrapped.  So 89% is already way better than I started originally with but at the same time 95% is  something that should be considered to be production ready like 95 and above so I want  to go further, I want to push the limits.  So what do I know about this guy again?  He is a student, isn't important?  Model actually yes because he has exam phase, he has some lecture phase and he has vacation.  And I know myself and I tend to behave completely different on exams and during vacation.  During vacation I slept like whole day through, whole night through.  So why should be different for him?  And in this case this is really artificial feature because it is static information taken  from the website of our university where I and he are students but you probably cannot  automate such information collection if you know how or if you trust the sources that  can provide you with this information.  Anyhow this is more or less static and now we have the four features model which performed  to 95% accuracy.  For me it was like 95, industry ready, I am done with it.  But if I take a step back and look at it, all I did is I tried to reconstruct behavioral  attributes or time attributes of the behavior of this guy and I started originally with  the only time stamp and I ended up with pretty good results only by feature engineering.  You should just believe me, you can obviously try it out yourself but tweaking the model  parameters, taking different models didn't perform well at all.  I never got better than 45% accuracy on only time stamps.  I tried STMs, didn't perform.  I tried some models from classic econometrics like Arima but this is not exactly the problem  I was approaching.  So let's check out whether this guy is awake now.  No he is not.  It's like three, yeah he will be probably awake in like two or three hours.  Ping me on the telegram or whatever just to ask whether he is really awake if you're  interested.  So this is pretty much what I just wanted to show you.  This is very simple yet very powerful and I've only used my, like I am an expert, I know  this guy, I've pretty much simulated my train of thoughts, what I know about the guy, what  I should, what information do I use and as a programmer I decided I tried to implement  or collect this data and yeah, that's everything based only on time stamp.  Anything if you have more data that you can unwrap.  Another case I can think of is for instance there is a competition on Kegel about taxi.  You have to compute the waiting time in the taxi based on the four or three years of taxi  time from Mexico or Ecuador countries and there are like normal data you can see in  the file they provide.  It's geocoordinates of pickup drop off, timestamp of pickup on drop off, the distance they actually  capped drove and how long did it took, how long did the whole ride lasted and stuff like  that.  But you can throw something like, I don't know whether the time, like for instance you  can, again can automatically unwrap the time stamps into date, sorry date, a day and an  hour and based on day and hour you can for instance decide whether it's a rush hour and  you know that in rush hour cars tend to have longer drive and waiting time than normally  because the roads are full.  Also you can define something like holidays, national holidays.  This is where really expert domain knowledge comes in and this is purely manual.  I don't really imagine anything comes close to this train of thought and performance by  using automatic approach because the data is really not there, at least it looks like it.  So it is costly.  It took me about three days to, like one day to build a model and one day I thought about  okay what can I do?  I only have this much data and only this data.  What should I do?  What helped me originally I've tried to, again reconstruct my own thoughts and I've tried  pretty much the model where I choose the fully connected neural networks is also reconstruction  of my brain work.  For instance for me it was important that today the feature importance is also took  some place and the layers they are, they're reflecting my for instance personal attributes  that I value more.  For instance I know that exams for this guy are not that important but what hour it is  is way important and you could see that after adding just day and hour we got 50% of accuracy  and by adding only season we got like 5 or 7%.  It is very error prone because it's expert domain knowledge you mostly especially in  literacy you will have to drag someone in who knows their stuff for instance some engineers  or even taxi drivers and between this communication over time between the expert and the programmer,  data engineer or data science some information will be lost.  Also again it is not really clear at what scale and what features should be incorporated  into the model or into the data set.  Sometimes you are so happy about the results about the progress that you do that you try  to throw in some features that are based on one each other meaning you will land into  a multi-colinearity situation where you have quite a lot of noise but not that much meaningful  data.  So sanity check is always a good thing just to create several features, think how they  perform, try to maybe fine tune the model, try to get the best out of this small scenario  this local optimal and then think again because every time you need to create a brainstorming  session you will be affected by the previous brainstorming session and at some point you  will run out of ideas.  So feature engineering is not a knowledge transfer it can be used as a part of it basically  you train models on certain amount of data that has at the first glance nothing to do  with your task but you have to train the first model somehow so it can be useful for that  step as well.  By unwrapping features you create even more features obviously and you can use feature  selection you can use automatic feature creation and what's ever because you have way more  space to do that.  I think it's very powerful.  You can argue with that but I think it's powerful.  And it worked with all models.  I tried some neural networks, I tried classical approaches like decision tree and they all  performed way better when they got more data.  It's like yeah, duh, but every time you see it in the code and the results it's wow.  So this is pretty much everything I wanted to say about the feature engineering.  Thank you for your attention.  So thanks for the great talk.  There's time for questions.  Hi, thanks for the talk.  I was just wondering have you at all compared automated feature selection to your insightful  feature selection and whether they perform better or worse?  Feature selection, like I won't use feature engineering when I have enough data to actually  use feature selection.  Feature selection comes in first when you have to, the data to search like to take some  features out to prepare for training.  Feature engineering is pretty much enable feature selection.  So I won't compare them on this level.  Thank you for the talk.  How did you select the frequency about your data, for example?  I was given it.  I was pretty much just shoved a file, a text file and I knew nothing more than this.  So in your case you want to know at what hour your friend is awake, not in what minute  or second or so?  It will depend on the task, obviously.  In this case it was enough.  Like certain minutes for humans to know whether somebody is awake or not, available or not,  or the real-time application would be an automated system for remote teams that will allocate  slots for meetings or some discussions based on their working patterns, where people tend  to be available, where you can use, for instance, multi-class models where you put something  like I want to do my correspondence, I am available for the talk, I am focusing on new  features and so on and so forth.  So in the real-time or real-world example you will probably need a different frequency  of the data.  In my case it was more than enough.  Again, based on the frequency of the data, different features will be more or less important.  For instance, exams, I was just lucky that I had two university phases in the data sample,  otherwise it won't make any difference.  Some more questions?  Hi, thanks for the talk.  You seem to be just advocating adding more and more features until you have improved  your model enough.  Is there some point that will start to become counterproductive and you will hit the curse  dimensionality by adding more and more dimensions to your model?  In my case it's rather hard because I honestly, for me, was already kind of, what can I also  add?  But if you start not with one dimension but more, basically, yes, you can hit the problem  where you're trying to get all the data simulating the whole real world, which is the point where  you start noticing some patterns or behavior, which we are talking about classification,  that is not typical for your case but is more general.  For instance, you know there are historical 100 cycles or whatever and there are not only  the amount of data important but things that show exactly when it happens, why it happens,  so you can learn from features.  This is where it starts to be counterproductive.  This is why I said define not more than four features.  First test them out, try to replace them.  It is a pretty small amount of work for the start of four features unless you have real  giant model but in this case you probably already not doing something wrong than you  started with the feature engineering and such a complex model.  After a while, I personally tend to say that depending on the case, how complex is something  you want to learn, if you want to learn just the sleeping cycles of a dude, then you don't  need much data.  If you're trying to build a self-driving car, oh boy, you're going to need quite a lot of  sensors, quite a lot of data, quite a lot of features.  So basically, depending on the complexity of the task, it depends the complexity of  the data and directly amount of features needed.  You can estimate yourself how complex your data should be or could be.  Reduce this by two and then start working.  Okay, any more questions?  Come on, I'm not biting.  Then maybe I have a question.  So I'm not a data scientist so I don't have much clue about this.  But I've heard that as you add more information to your model, there's a risk of overfitting.  Did you do some sort of cross-validation?  Because your precision goes up.  But that's sort of too expected anyway, isn't it?  Yeah, true.  In this case, I didn't because I had way too less data and I was in the territory of always  underfitting.  The amount of features was totally under control.  If I would start to throw something like, does this guy live with the parents?  Does he has a pet?  And so on and so forth.  And probably I will overfit at some point.  So basically, in order to not overfit or to have low risk of overfitting, you need to  not only keep your data, you need to keep your data still low dimensional.  To not overcomplicate the life or description of the situation.  Okay.  One last question.  There's still time, but...  What if we come up with a feature that is really not helping with the accuracy then  we'll just drop or how can we observe that?  There was a thing.  When I initially started to creating features, I also thought about, I didn't thought at all,  at the first, about the scale.  But this was okay because I had like four categories.  But at some point I tried to scale them.  For instance, you saw here this three features, two three features falls.  This is basically scaled and not like row version from zero to whatever.  And this is more of a normalized one from minus two to two or so.  And this models actually perform not way better.  Not always.  But I tried to define season like minus two, minus one, zero, one.  And it decreases the performance by seven percent.  Also some features, as I said earlier, are just a bit different form of already existing.  So they won't really decrease your performance.  They might have, but they won't help either.  Also always check your data whether this, the feature you want to create will be present.  I had another case where I was really interested whether some national holidays are affecting  taxi waiting time.  I don't know, I spent not that much time, like four hours, by really finding out all  the information and turns out they just don't have those sample points.  So time wasted.  Okay, one last question.  Thanks.  So when you're dealing with these small data sets, low dimensional data, how do you differentiate  between the cases when there just isn't enough information to build a useful model and when  you just haven't found the right manly engineered features yet?  Mostly gut feeling and common sense, if honestly.  My differ, again in this case, it was pretty obvious.  I simulated my own thoughts about the problem and I figured out what I personally call feature,  what my brain defined as a feature.  And this is pretty much where I stop.  In more complex situations, I probably would start with model first and not the data if  I have at least like four features.  And after a while, I will notice that no model tuning actually works.  And but the data is clean.  I have, I don't have missing values.  It is sane.  And then I will ask what my model wants to describe.  And as a human, I'm smarter than the model because I knew more.  I've seen more.  And this is where I will start trying to dig deeper and bring some features in that model  might find sufficient, might not.  Thank you very much.  Okay.  So give a big hand to Alicia again.
Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_0_995.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_996_1979.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_1980_2999.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_3000_8327.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_8328_8387.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_8388_10127.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_10128_11891.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_11892_12071.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_12072_12251.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_12252_12827.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_12828_12875.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_12876_12923.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_12924_13127.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_13128_13175.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_13176_13643.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_13644_13703.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_13704_14003.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_14004_14051.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_14052_14687.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_14688_15287.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_15288_16823.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_16824_17135.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_17136_17975.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_17976_18707.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_18708_20855.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_20856_20903.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_20904_20999.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_21000_21059.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_21060_22763.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_22764_26579.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_26580_31907.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_31908_32063.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_32064_42779.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_42780_42947.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_42948_43283.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_43284_43331.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_43332_43463.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_43464_45023.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_45024_47471.png Alisa_Dammer_-_Data_is_not_flat_YljaTn-CAP0_47472_47578.png