Real-time Face Detection and Emotion/Gender classification with Convolutional Neural Networks
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Arriaga, Octavio
In this work we present a real-time system for face detection and emotion/gender classification using Convolutional Neural Networks and Haar-like features.
 Well, thank you all for coming to my presentation on real-time phase detection and emotion gender classification using convolutional neural networks. First, a little bit about me, Octavio. It's a, I am a, yes, as he mentioned, master student here and well, participant of the Robocop athlete competition. So all of this work came or was brought up to because of the Robocop, international Robocop competition for the, at home leak. So yes, well, our main motivation, what would we, why would we like to have an emotion and gender classification system? Well, here in our university, we have this robotic curse and obviously our intentions are to put this into a robot. And well, domestic and our current robot platforms are currently being used for home systems or as personal intelligence systems. So in this case, we have the robot here Pepper, which is very famous. And I believe this robot here belongs to a French company as well that they're trying to sell it so they can deploy it at your home and they can be, serve as intelligent system that can be, take pictures and remind you of things. Yes, obviously these sort of robots require a high level perception skills and in this case, we can think of it, we can think of several examples. One of them is a person that is an elderly woman or man that is just trying to communicate that the person is in pain or that the person might have some sort of, some sort of anger or might be sad and maybe it's unable to express it. So a person might be in pain that is unable to express it or therefore the robot might, we would like that the robot extracts information from this person's face and in order to proceed as it should proceed. For example, in order to call someone or in order to tell someone. And yeah, there are also other cases in more going to the other extreme in which the robot would like to take a picture and there we would only like to take a picture when everyone in the frame is happy, right? So I mean, there are a lot of cases in which we can think in which we would like to extract information from the user's face in order to convey some sort of movement with the robot. And well, one important characteristic of emotion classification is that it's not so easy. One would think that maybe classifying someone else, someone else's emotion only by looking at the face could be potentially or substantially easy but apparently humans perform around 65% accuracy on the on this current data set that we're using and which is the third 2013 data set. And yeah, it's open source and everything and it has 35,000 grayscale images. So in this example, I have chaff all these images and then we can look around and see, yeah, I don't know, like it's easy for us to take this person that it's happy, right? But maybe this one. Yeah, I don't know. I don't know which class should I assign to this person. Maybe SAT or neutral and also, I mean, the baby, like, I don't know which class to give to the baby, right? Which level to give to it. And yeah, another thing that we could see from the data set here is that the faces are not presented in a canonical way. That means that the pose of the face is rotated. So this might also be problematic eventually for performing a classification. And yeah, the other data set that we use was the IMDB data set which consists of images from the IMDB website. It's around almost half a million pictures. And as you can see, there are pictures of the small world Hollywood ecosystem, right? And these eventually leads to problems as we will talk about later. But yeah, it's one of the biggest data sets out there which has around half a million images of genders or sex. And yeah. So in spite of the problematic things that we might able to have in our robot, which would be basically that we are constrained hardware-wise and also that the estimation or the classification of gender might be and emotions are rather complicated, we have to create a system that is able to be efficient and that is able to perform all of these tasks. So in this work, we present such system that is able to do phase detection, gender classification, emotion classification all in real time and open source. We release our models and our code open source. So it's able to also, it's robot or framework independent. So we also try to make the community or the robot community try to adapt our software. So we try to make all the appropriate software accommodations so this can be implemented as well. And furthermore, we also know that convolutional neural networks or neural networks in general are used as black boxes. And I mean, there are black boxes, but often we would like to see what are the hidden features learned by the convolutional neural network, therefore we also implemented a visualization method which I'm not saying it will tell you what actually it's the convolutional neural network learned, but it tells you something, it makes a visualization that is, I believe, interesting to interpret. And this call is, this solution is called guided back propagation. So I don't know how you guys are familiar with neural networks. I worked for around one year, but I don't know, could you guys raise your hand if you have worked in neural networks before so I just can get a feeling? Yeah, okay, so there's a lot of people here. So I will go quickly about it, but not too fast. So basically, neural networks are functions approximators. So you have some inputs and you have some outputs. And you want to create a function that maps these inputs to these outputs. And how you want to create this function, it's by a set of models in which the model contains weights which are represented by these arrows. And we are changing these arrows using an optimization algorithm that tries to minimize an error at the outputs. And yeah, so basically trying to minimize an error and then we change the weights accordingly in order to make these neural network approximate the function given by our data. And here I display two of the most important separation of neural networks. One of them is recurrent neural networks and the other one is fully connected neural networks. Recurrent neural networks have a hidden state. That means that the neural network itself is dependent on the history of the previous inputs. So one could think that recurrent neural networks might be better or more suitable when inputs are given sequentially. So for example, time series is one sort of area in which recurrent neural networks are used to, but also language. So for example, language is also we construct a sentence and a sentence has a sequential meaning to it. And yeah, and the work here is, well, convolutional neural networks not necessarily feedforward or fully connected neural networks. So I will explain a little bit about convolutional neural networks as well. Convolutional neural networks are basically feedforward neural networks with two imposed constraints. And basically they substitute the matrix multiplication that was happening here for a convolution operation. And I will keep in detail some explanation about the convolution operation. But here we, in the convolution operation, we have a kernel and this kernel gets cobalt in our image. So here we have a picture of our hoaxula. And we will make a convolution. I don't know if you guys are familiar with the operation, but it's a, you just put this matrix over here and you multiply all the elements and just sum them up. And this gives you one pixel over here. And then you keep moving forward this sort of operator and you'll get this sort of pixel values that each time this operator is applied one pixel here. And at the end, from this handcrafted kernel, I will get some edge detection. And this edge detection will help us eventually, or could help us eventually to create a system that is able to recognize something, right? But the idea of a convolutional neural network is to know or to learn the elements of the kernel. So in order to, when it cobalts the image, you get features that are important or that are more suitable for the classification. And as I mentioned before, these are the two constraints that are given to a neural network or to fully connect the neural network in order to convert it into a convolutional neural network. One of them is local connectivity, which tells you that not all the input values are going to be as important for the neural network or that you're only looking at the local patch of the input values in order to make explicit computation. So or in more, in our words, a pixel that is located on the far end of the image might not be as important for a local patch that is located on the other end. And weight churn, weight churn tells you that a set of pixels or a set of extracted feature from a local patch of pixels will be important for the other patches. And therefore, you will assign the same weights to all the patches. And at the end, you impose these two constraints. And yeah, magically, your feed-forward neural network is converted into a convolutional neural network. And this is one of the images that I like very much because it explicitly tells you how to calculate the or how the convolutional neural network works. But in this case, let's think about an RGB image. An RGB image is a matrix that consists of or three matrices are stuck together. And we would have a kernel, which is represented by its other tiny matrix, which is also as a volume of three stack matrices. And what we do basically is just cobalt the kernel with the input. So we do this by, for example, looking at this specific x, y in the input feature map. We will multiply each value d0 times d0 and this one times d minus 1. And we will do this with the first slice of the kernel. And then the second slice of the feature map, we will do it with the second slice of the kernel and so forth for the other ones. And then once we have computed all these multiplications, all of these will get sum. And then finally, all these three values that came into scalars will get sum. And then finally, avias will also get sum. These summing values at the end are very magically given and also with this something that was improved later on, which I'll talk about as well. But yeah, this is the way in which a convolution neural network looks like in the convolution layer at least. So it's just you're learning these sort of 3D kernel weights that are cobalt in another volume of an image, for example. And another interesting thing is that we have, for example, here two kernels, kernel 1 and kernel 2. And kernel 1 gives us another feature map and kernel 2 will give us another feature map. That feature map is just a modification of the original image by cobalt in the kernel. So if we have, for example, n or n feature map, sorry, we will have n or sorry, if we have n kernels, we will have n feature maps at the end. So we can control the amount of feature maps you want by the amount of kernels that you have. And this is one of the most successful convolutional neural networks, which is BG16. I believe, one of 2014, the ImageNet competition. ImageNet competition is a competition of classifying 1,000 classes from 1.3 million images. And this is very incredible result. I like to think of it as some sort of system that you can integrate into, for example, into your robot platform and is able to identify 1,000 classes. So now we could put this into a robot and the robot will walk around and you can take picture of something and it will say, yeah, this is a computer, yeah, this is a person, and he will do this for 1,000 classes. And now we can do this. And this is very easy to do. But yeah, these sort of convolutional neural networks are computationally very expensive. For example, they have around 138 million weights. But another interesting thing is that, as I explained before, we have these convolutional layers. And at the end, you add a fully connected layer, which is a very traditional approach to construct a neural network. So you have this convolution, convolution, convolution, convolution, and then you have the fully connected layers. And these fully connected layers account for the 90% of all parameters in the network. So this is also very, very interesting. Like, it might not be completely convolutional, but it's 90% some fully connected network, right? So, yeah. So another solution to, in order to reduce the amount of parameters that we have in our fully connected part, is to create or to add a global average pulling layer. And a global average pulling layer will take the feature maps, and it will just make them sorry, it will take these feature maps and it will convert them into a single scalar value. So in the previous approach, what was happening is that we have the feature maps, and each feature map gets flattened and then concatenated with all the other ones. And then from here, we will perform a fully connected layer, and then it will turn out to be the output values. But in this case, what we are actually going to perform is we're going to take this feature map and we want to do a global, we will perform an average of all the elements in the feature map, and then this average will turn itself into a scalar value. And then these scalar values will actually be the output nodes. So why are we performing the average? This is something that is rather interesting because average gives you information, you need information of all the single nodes in your convolutional neural network in order to have something significant. But if you have something like the maximum value of it, then you're kind of losing information about it. So global or the mentality of why you think average is just to give individual representation to all the nodes in the feature map. And yes, our initial architecture, which is the one that I proposed in this work, it consists of nine convolution layers which use global average pulling at the last layer, and it has around 600,000 parameters. And it achieves an accuracy of 96 in the IMDb data set and 66 on the third 2013 data set. This model is 7.5 megabytes, so it's rather small. And yeah, other methods achieve the same accuracy, but using an ensemble of convolutional neural networks. So this also might prove slow for real-time systems. But this is not the end of the story here because at the end, 600,000 parameters seems rather cumbersome for a simple task. So we decided to explore more on modern architectures on convolutional neural networks, and we encountered this exception architecture, which combines two of the most successful experimental assumptions in convolutional neural networks that is the use of residual models and depth-wise separable convolutions. And the use of these two models or assumptions are basically the state of the R-Rine and convolutional neural networks. Which is the residual models that instead of actually performing a sequential operation on the future maps, we will divide the future map so it adds an identity to it. And we will perform a convolution again by a nonlinear function, and then we add the previous future map. So this is something like a perturbation method in which you have the future map, and you want to see which perturbations in the future map actually make it better in order for the classification to be more accurate. But the most important thing about the residual models is that it makes the back propagation algorithm which changes the, which is optimization algorithm or which is the algorithm that modifies the way sufficiently, makes it easy to create a bridge between the previous future maps. So it's able to propagate the gradient easily through the network, which is this is also a well-known problem in convolutional neural networks. That the convolutions get so, or the neural network gets so big that the actual back propagation of the gradient also gets decreasing or also explodes depending on the uncertain assumptions. But this allows you to back propagate the gradient easily, and therefore this architecture one, I believe the 2015 ImageNet, which consists of 101 layers of convolutions, one after each other. And the other one, which is that why separable convolutions. And as I explained before, you have these sort of kernels, right? These 3D volumes that map the input images with the, that try to convert it to another set of future maps. And in this case, the depth-wise separable convolutions try to separate the cross-correlations between the spatial values and the channel values. So instead of having 3D volume kernels that will be cobalt in the input feature map, we will make them channel dependent. So in this case, we have a channel that has its own kernel. So all the input feature maps will have a specific kernel, which is, yeah, in this case, 8k times 1, and we will learn to mix the values outputted by the filters in several ways, which are given by this one-time-one convolution. So the one-time-one convolution, what it's doing, it's mixing the values given by these sort of kernels here. And yeah, this sort of reduction gives you a factor of 1 over n plus 1 over d squared. And this also eliminated some parameters in our convolutional neural network. And at the end, our final architecture, which is based on this exception network, was able to, yeah, we reduced the parameters so 10 times more. So we have 60,000 parameters now, and the model is around 853 kilobytes. It achieves almost the same accuracy in the generic classification in the previous model. If you recall, it's 96. Here's 95. And also, emotion classification was exactly the same, which is 66. And our complete pipeline that included the phase detection and the generic classification, emotion classification takes around, yeah, I mean, this much of seconds, 0.48 seconds, right? And this on a low-end GPU, and yeah, on an I5 CPU was around 0.051 seconds. So I mean, it's a... So basically, we reduced the amount of parameters that we actually need in a convolutional neural network in order to perform a system, a real-time system. And these are... I'll show this later. These are some of our results. So this is what we want, right? We have...this is our Robocop team in Italy, and we're performing a competition. So here we have blue represents women, and the other 100% men. And yeah, happy men. I mean, this is very fairly easy to represent, right? Happy, happy, happy, happy, happy. This is neutral, if you cannot read it. And yes, we have also something here, which is SAP men. But if you...I don't know, it's a...this is...this error here was made for the...because of the phase detection algorithm, which is the open-CV implementation. But yeah, if you can look at it more closely, you can see there is some sort of eyebrows there, and yeah, the map looks at, you know. And this is another fun thing that I work on, which is...yeah, this is the Solvai conference, and here is Moudicouli, and then Albert Einstein, and yeah, the other famous physicists, and it says angry women, which I believe is correct. And yeah, neutral men. This movie that I like, which is called 12 Angry Men, and then you can see the other all of them are angry. I think none of them are angry. It's neutral, sad, neutral. And yeah, I'll show you some of the results that I have in our Github repo. So, yeah, by the way, this is...these are positive that was...yeah, that I did in order to put my project and everything, and yeah, this is rather interesting because I published it without any idea of like how interesting for people could be, but it turned out to be that a lot of people like it, and yeah, I got a nice contribution from the Opens of the Community, and here are the same examples. Well, here's different faces for the emotions, and here's a real sense of me, and I'll explain later why this might be interesting, because like, okay, this is sad. I might not look so sad, right? But this is angry. I'm not angry here. This is kind of weird, right? I'll talk about my errors later on, which is also very interesting. And yes, perfect. And yeah, and well, let me just go back here, which is the visualization technique that I was talking to you about. This visualization technique is called guided back propagation, and basically what it's doing, you keep as input the image, and then you wobble the picture, or you can think of it as wobbling the elements of the picture and seeing which neuron gets activated in the neural network. And you look at the neuron that gets activated the most, and then you keep wobbling the pixels, and you find the pixels that correlate more with the activation of a neuron inside your convolutional neural network, and therefore it could tell you which pixels in the image make a neuron activate the most. And here, for example, I believe every row is an emotion, so apparently this guy was angry, because yeah, I know Samuel L. Jackson was angry here. And so we can, for example, see that the neural network responds very heavily on the smile, obviously, so this is something that, I mean, we expected it from the neural network. We also see something interesting here, which is that the surprise classification depends solely on the eyes, or mostly on the eyes. So how big your eyeballs are open. So this is also very interesting. And also, yeah, angry, which is the part that I wanted to talk about. Like, angry seems that you only have to like really, I don't know how this, you only have to make your face like this, and you will trick the neural network to believe that you're actually angry. So you just have to do this. Not really, you don't have to be extremely angry. You just have to really perform this sort of face. And this is why I was telling you in the real-life model that I didn't look angry. But yeah, this is why you have also to be careful around these open source implementations, right? Because I know what activates the neural network the most, so I can make the faces that activate it, right? So you also have to think about it. And yes, another interesting idea or problem that arose is that most of, as I mentioned before, yeah, this dataset, right? I mean, this dataset, it's rather very oriented to Western-looking faces, right? So I was here in the Hoshul and the Robocop competition what's happening in Japan, and I received a lot of emails from people in Japan telling me that my implementation didn't work because it was not working for Asian people. I mean, it was not working because obviously the dataset is of these people, right? And also something very interesting that I encountered was the use of glasses. The neural network confused people that got angry with a person that was only wearing a glass or wearing glasses like mine, for example, which are very dark. But also more, most interesting, I think, is that the neural network got conf... I had a friend of mine and she was sitting in front of it and we had the live demo on it. And she didn't have any glasses and it was classifying her as a woman. She put her glasses on, it said men. So this is something also very interesting that I would like to pinpoint from the dataset. I mean, you really have to dig into the dataset to look what errors my... what errors my occur inside the network, right? It might be biased to say that because a person is wearing glasses, it might be a man. Why? Because all the samples that you give to a neural network of persons wearing glasses are men, right? And... Yeah. Yeah, so future work. The reduction of parameters made us create a real-time system, but this is not the end of how much we can reduce the amount of parameters. We also can perform some evolutionary strategies in order to reduce the parameters, depending on maybe the amount of kernels that we have. This was also not a parameter that I tweaked too much. So I believe further optimization can be developed here. Also we can incorporate more classifications. I mean, the networks that we are turning are so small that we can create more of them in order to produce, for example, H. And this also H data is in the IMDb dataset. So we can easily train another neural network for this. And then the last one, we should create double-headed models, which means that from a single forward pass, you can output several classes that... several... yeah, several classes that include, for example, Hapi or that include emotions that include gender and that include H. But in order to produce this successfully, we will have to have data set that contains the tree labels. So that every time we forward pass the image, we have the information of the tree labels. So this is also something that we do not have and this is something that, therefore, I couldn't able to make it. But at least for the H and for the gender that are contained in the same dataset, we can create this sort of double-headed neural networks. And yes, thank you for your time. And I will... how much time do we have? Is it... we have time? Yeah, okay. This is the first... this is sort of half of the presentation that I was trying to give today. And if we have time after the questions, I would like to give another talk if you guys are interested, which is again, convolutional neural networks and offers... yeah, and opens over a relation to it, which is for image captioning. But yeah, thank you. Any questions? So Chris here read an article about recognizing street signs, just the same sort of neural networks. And the question is, how do you know that, okay, we have a stop sign and put some extra text on it. And the turning sign and the stopping sign. And the movement, you know, if it's an octagon and it's red and the stop sign will let the text on it. It can be an error or whatever, you know, it's a stop sign. How can one assure that one of the strings is such a thing that really the most important things are used for this termination and not something that might work for the image set but may not work for the real life? Thank you. Yeah, this is a very interesting question, which is how can we make sure that our networks are able to generalize correctly and that the small perturbances in the input data don't change the output. And this is a very hot topic right now. And I believe maybe the paper that you read was on adversarial networks, which they tried to create different modifications. So I think there's even some examples in which they add noise, which for us, it's, we cannot see this noise, one, it's an image. And then the class changes entirely from what it was assigned, for example, into a panda. And but the noise is, it's not, we are not able to perceive it. And this, yeah, this is a well-known problem. But another thing is that it's well-known for convolutional neural networks, but it's also well-known for most of the machine learning algorithms. So it's not only for convolutional neural networks. So one can also optimize noise that you give to input data to an SVM or something else that will change the output. So I mean, this is, this big problem in machine learning in general and not necessarily only in convolutional neural networks. I think from the F2P such an example, we want to make sure that it recognizes the box and that it recognizes the color red. And only if it's a two or three, it takes it to the stop sign. So it's much more. I mean, yeah, the question is that does it really use that black box or does it really see a train into digital business and then find a rule set how to combine these signals? Yeah. I believe what people... I mean, currently there's, as I mentioned, this is a very hot topic and currently there's even a competition in Kegel in which they're trying to create adversarial examples so in order to make the neural networks more robust. So there's no answer as far as I know right now. No? No? There's this onegravel that's kind of is your useable role. What was the mechanism you trained in your network that would help you pick different kernels and help them to access rates to the given information of age or main email information? And you picked the most successful set of kernels? Could you tell everyone what was the input set to secure that for energy? The training process or the evaluation process? Yeah, okay, so the question is how is the training and evaluation process performed? Well, the training process is, I believe, well known in neural networks, which is just the implementation of the back propagation algorithm, which that's why it's so important because I don't have to choose the elements in the kernel. The kernels or the neural network can learn to choose the values for me. And the evaluation procedure, well, we basically divided our dataset into training, validation, and test, and then we just performed, we used the model that performed better on the validation, and then we just proceeded to test it. Yeah, so the complete, chosen of the values in the kernel is not my idea and do it. The optimizer did it for me. Okay, so what we measure is the end-to-end data set and what the network is. Exactly, yeah. But how do you divide between just trained following simple tests, over trained, and some under trained and how do you get the measurement for this? Is it just the comparement to the end-to-end data this image success? Yeah, you have to perform the, this is why you have the validation set because you want to see that your validation set is performing correctly, right? So you split something that you're not training on and in order to see that it's performing well and in order that it's able to generalize well. And yes, this validation set comes only from the original data set, yes. In order for back-to-back-propagation to work, you need to define the error function. Yes, exactly. Define for this case because there's multiple things that you need to account for. Did you test for a matching protocol for face-to-face with this, or matching for where it faces the image and then how it can be used? So you have a distance measurement for different features of different specifications. Yes, the loss function is nothing fancy here. It's something, you have several loss functions as you mentioned, you have this square loss, for example. But here we only use the typical classification loss entropy loss, which is just tell you the entropy of how well, or the cross entropy basically. So it tells you how well your output values in this looking at them, if there were a probability distribution, how well it compares to the probability distribution of the actual thing. So the cross entropy loss, yes. So nothing really special. And we're now just doing research in analyzing the videos of those images. Yeah, so the question is whether or not we can incorporate video analysis in our robots. And yeah, obviously this is something very interesting as well. But this is also something that becomes computationally complicated, right? Because you have to account for now video, we'd add another dimension to your neural network. And, but yeah, there's current research on this, for example, how you call this, to detect whether what a person is in actual classification, yes, actual classification. Yeah, further questions before I proceed to the next slide? Yes. You mentioned the reduction. So you mentioned that the fully connected layer has accounts for 90% of all network weights. But can you name the numbers of the reduction? So how much weight do you manage to cut in your examples when you're planning it, one by one, and the solutions and everything? How much did you win? Yeah, this, the question is how much did I win from cutting out the last fully connected values? Well, I would have had to train a fully connected network in order to see how many values are actually, how many values actually give me the same classification accuracy. So I didn't do this. I proceeded explicitly to perform the global average pooling, which I knew it was something that already gave you a reduction. But as we saw, for example, that incorporation of the exception architecture, it gave us, yeah, it reduced 10 times more, right? Yes? So as a concept, I've seen a mention of, for example, you want to implement some neural network and it's really hard to train on your hardware. So one of the solution is to take, for example, an Alex Nats, get rid of the fully connected layer, freeze all the convolutions and everything, and just train this last fully connected layer to, let's say, detect only Nats. So I didn't see this fact about 90%. So is it the hope to use this sort of approach to train something easily in home conditions on your GPU? Have you ever tried something like that? Yeah, so the question is have I tried doing fine tuning? Yeah, fine tuning. Oh, yes, that's the word. Yes, I've tried doing fine tuning. But as you mentioned, as you saw in the BE-16 network, as you saw in the network, the network consists of three fully connected layers. So you don't take out the three. You just take only one. And then from on top of this one, you train. Yeah, so you don't get rid of this 90%, right? Yeah, you still have some. Cool, yes? Will the slides be available? Yes, the slides will be available. Sorry for that. Yes. OK. Where? Would you, is this information somewhere already? There is information of my, of everything here on the, on my GitHub repository. So all the information you can find there. But I'll make sure that also people from Frost can have it. OK. Good. Then I'll proceed to the, I mean, if we have some time, I guess, we have 20 minutes. I'll proceed with something that might be also interesting for people that it's on. So, yeah. Yes. So, what I'm going to present now is image caption and classification of a normal situation. This project was also made with Professor Paul Ploeger. And it's also trained a convolution and an LSTM network in order to account for situations in which the situation, in which values might be dangerous. Or for example, we would like to train a neural network that tells us that creates a description of the situation. And this description will give us information on whether or not the situation should be important for the robot. So image captioning consists of a convolution neural network and a LSTM network that in conjunction perform or construct a sentence of the image. So the neural network will take the image and it will perform, it will say a group of young people playing a game of frisbee. And this is image captioning. And our basic example or basic thing that we would like to perform is a system in which a robot sees a situation. It takes a picture of the situation and then it describes the situation or it makes a sentence of the situation. This sort of sentence can be communicated to someone else and then they could proceed in order to, as they see fit, regarding whatever is happening. So for example, in this case, the robot will approach and instead of having several components in a robot in which they try to do maybe post-estimation in which they try to do several classifications, we will have a single component that will tell us everything that we would like to know about the situation at hand. So in this case, the robot will approach, it will take a picture and then it will say an elderly man is laying unconscious on the ground. And this is the system that we created. And, yeah, LSTMs are another form of recurrent neon network that are a little bit more complicated. But, yeah, they are differential versions of the memory chip in a digital computer. This is what Alex Reb said, which is a famous researcher that worked with LSTMs for a long time. And basically, LSTMs contain several gates that control the input, the output, and the hidden state of the network. And I will not go into too much detail about this, but, yes, the process in which the image captioning system works, it just passes an image through a pre-trained convolutional neural network, as I explained before. It poses this into the first state of the recurrent neural network, and then the recurrent neural network will start predicting, give it the information of the image, a word, or the first word in the sentence. Then it will use the next value in which it was trained on, and the hidden state in order to produce the next word in the sentence. And it will continue to carry on until it performs a sentence such as this. A group of young people playing a game of Street Street is explicitly given here in which you have a stored token, you have the information of the image, and then it produces a straw. And then given the word straw, and the hidden information of the previous step, it will produce hat. And then if you give hat and the given information from the previous two steps, it will give you the token end. So we'll learn to create a sentence. And this, there was a problem regarding our data contained in the, for this case, because there are usually not data sets that contain anomalous situations. So we don't have data sets that contain, for example, guns or people maybe in pain or blot. So we had to actually go into Flickr, look for these images, and then ask people around our university to label or to create a sentence for each of these images. So this was a very enduring process. But at the end, we collected 8,000, or sorry, 100, 1,000 and 8 captioned images. And that were captioned by 20 different people here. And yeah, all of our images are on the creative common license, and I hope to make them available soon so people can work on this also in machine learning. And these are some of the examples that are recollected in our data, which include, for example, a person that has been injured, a police officer holding a gun, and some violent images and some things running on, being on fire on the street. And we implemented one of these image captioned models, and we obtained a score of 14.2, a meter score, which tells you, which tries to compare sentences, basically, how much a sentence is in accordance to another reference sentence. So this sort of evaluation is also very mysterious, because it's built by humans in which some parameters are also well tuned, and I believe this metric should improve. But anyhow, this is the metric that people use in these captioning models. And we also obtained an accuracy of 97%, which means that I'll give you an image, and then I will tell you, I want to know whether the image is an anomaly or not an anomaly. So we received an accuracy of 97% for this, and these are some of our results that we obtained. So this is the image that we received, and the neural network said this, man is doing a trick on a skateboard. Then we gave an image of this, a train traveling down these tracks next to a lost room field, a group of people flying cats in the field, pizza with cheese and herbs on the table, this with a laptop and a monitor. This is also, I think, very difficult and could also be a good application for robots. And then a man on a server riding away in the ocean. And these are some of the errors that we also got from these non-anomalies images, which are a woman sitting on a bed with a laptop computer. Well, it's not a laptop computer, but it's some sort of cushion. And then a dirty toilet in a small room with a wooden floor. It's rather dirty, but if there's no wooden floor or anything like that. But there's wood here. So I don't know what's happening actually here, right? So it's also very, we would have to make some sort of error evaluation of the network to actually see what's happening there. A cat sitting on a car seat with a concerned look. And this work here, concern or concerned look, it's also very like, why did it happen, right? Well, I went into the data and I found out that a lot of people describe images that are looking towards them as concerned. So this is why we have the word concerned here, because it's also biased in the data, right? And then we discovered that a neuron doesn't know how to count. So a cup of coffee and a cup of coffee. So this might not be entirely wrong, but still it's a man is eating a sandwich and drinking beer. And a woman holding wine glasses. So this is also wrong. And these are some of the examples that were, that were, that we captured, that our neural network captioned and that I believe that they're positive. So a man is holding a gun. So it only took the image and was able to create, learn the English language and then create a sentence out of it. And then here, house is burning. A car is crashing on Snow Corbett Street. There is a woman with blood on the floor. A man is choking on our man. A firefighter is trying to put out a fire. So I mean, this is what we wanted. We wanted the system that we can incorporate in a robot that is able to tell us whether some information might be an anomaly, right? And these are some of the results in which, yeah, in which the neural network didn't perform as well for anomalies. People with red helmets are sitting on a car on A. So we see that the neural network doesn't always finish the sentences. There's a broken window laying on the ground. I believe that this might be because there's some variation here on the floor, but I'm not entirely sure. And a man is showing his injured shoulder. So it's not showing his shoulder. And then a man is showing his right arm in which he has a severe injury. I mean, this is not his right arm. This is his leg, but it's difficult, right? I mean, a man is being held by the police. So I mean, it's not a police, but referees. So I mean, a man is dancing with a woman on a dance floor. I don't know what's happening there, to be honest. So this is also a difficult caption, right? Like, what are we asking to the network? What are we training it for? And this is also complicated, right? So we have to look at the images we're giving. And, yeah, thank you for your time. This is an honor small project we had. Thank you. Questions? Questions? Yeah? Yeah. Yeah, the question is whether I use, whether I write, whether I wrote everything from scratch or whether I use a framework? Yeah. I use TensorFlow with Keras, which is the easiest way. Yeah. Another question? Yeah? Exceptional events data sets pretty small just for thousands. Yeah, it is rather small, yes. Aren't you concerned about forbidding more just training of that particular disease objective test actors? Yes, no, we perform, yeah, we divided the data set accordingly like 2020, like 80% for training, and then from this 80 and our 20 for validation, and then we did the appropriate values so we don't overfit it, at least in the classification. But the captioning part is rather complicated to evaluate. And this is why I was making a small remark on the metrics used on the captioning scenario. It's also very hard coded, so I don't trust it too much. Yes? Any other question? Yeah? Okay, cool. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 Well, thank you all for coming to my presentation on real-time phase detection and emotion gender  classification using convolutional neural networks.  First, a little bit about me, Octavio.  It's a, I am a, yes, as he mentioned, master student here and well, participant of the  Robocop athlete competition.  So all of this work came or was brought up to because of the Robocop, international Robocop  competition for the, at home leak.  So yes, well, our main motivation, what would we, why would we like to have an emotion and  gender classification system?  Well, here in our university, we have this robotic curse and obviously our intentions  are to put this into a robot.  And well, domestic and our current robot platforms are currently being used for home systems  or as personal intelligence systems.  So in this case, we have the robot here Pepper, which is very famous.  And I believe this robot here belongs to a French company as well that they're trying  to sell it so they can deploy it at your home and they can be, serve as intelligent system  that can be, take pictures and remind you of things.  Yes, obviously these sort of robots require a high level perception skills and in this  case, we can think of it, we can think of several examples.  One of them is a person that is an elderly woman or man that is just trying to communicate  that the person is in pain or that the person might have some sort of, some sort of anger  or might be sad and maybe it's unable to express it.  So a person might be in pain that is unable to express it or therefore the robot might,  we would like that the robot extracts information from this person's face and in order to proceed  as it should proceed.  For example, in order to call someone or in order to tell someone.  And yeah, there are also other cases in more going to the other extreme in which the robot  would like to take a picture and there we would only like to take a picture when everyone  in the frame is happy, right?  So I mean, there are a lot of cases in which we can think in which we would like to extract  information from the user's face in order to convey some sort of movement with the robot.  And well, one important characteristic of emotion classification is that it's not so  easy.  One would think that maybe classifying someone else, someone else's emotion only by looking  at the face could be potentially or substantially easy but apparently humans perform around  65% accuracy on the on this current data set that we're using and which is the third 2013  data set.  And yeah, it's open source and everything and it has 35,000 grayscale images.  So in this example, I have chaff all these images and then we can look around and see,  yeah, I don't know, like it's easy for us to take this person that it's happy, right?  But maybe this one.  Yeah, I don't know.  I don't know which class should I assign to this person.  Maybe SAT or neutral and also, I mean, the baby, like, I don't know which class to give  to the baby, right?  Which level to give to it.  And yeah, another thing that we could see from the data set here is that the faces are  not presented in a canonical way.  That means that the pose of the face is rotated.  So this might also be problematic eventually for performing a classification.  And yeah, the other data set that we use was the IMDB data set which consists of images  from the IMDB website.  It's around almost half a million pictures.  And as you can see, there are pictures of the small world Hollywood ecosystem, right?  And these eventually leads to problems as we will talk about later.  But yeah, it's one of the biggest data sets out there which has around half a million  images of genders or sex.  And yeah.  So in spite of the problematic things that we might able to have in our robot, which  would be basically that we are constrained hardware-wise and also that the estimation  or the classification of gender might be and emotions are rather complicated, we have to  create a system that is able to be efficient and that is able to perform all of these tasks.  So in this work, we present such system that is able to do phase detection, gender classification,  emotion classification all in real time and open source.  We release our models and our code open source.  So it's able to also, it's robot or framework independent.  So we also try to make the community or the robot community try to adapt our software.  So we try to make all the appropriate software accommodations so this can be implemented as  well.  And furthermore, we also know that convolutional neural networks or neural networks in general  are used as black boxes.  And I mean, there are black boxes, but often we would like to see what are the hidden features  learned by the convolutional neural network, therefore we also implemented a visualization  method which I'm not saying it will tell you what actually it's the convolutional neural  network learned, but it tells you something, it makes a visualization that is, I believe,  interesting to interpret.  And this call is, this solution is called guided back propagation.  So I don't know how you guys are familiar with neural networks.  I worked for around one year, but I don't know, could you guys raise your hand if you  have worked in neural networks before so I just can get a feeling?  Yeah, okay, so there's a lot of people here.  So I will go quickly about it, but not too fast.  So basically, neural networks are functions approximators.  So you have some inputs and you have some outputs.  And you want to create a function that maps these inputs to these outputs.  And how you want to create this function, it's by a set of models in which the model  contains weights which are represented by these arrows.  And we are changing these arrows using an optimization algorithm that tries to minimize  an error at the outputs.  And yeah, so basically trying to minimize an error and then we change the weights accordingly  in order to make these neural network approximate the function given by our data.  And here I display two of the most important separation of neural networks.  One of them is recurrent neural networks and the other one is fully connected neural  networks.  Recurrent neural networks have a hidden state.  That means that the neural network itself is dependent on the history of the previous  inputs.  So one could think that recurrent neural networks might be better or more suitable when inputs  are given sequentially.  So for example, time series is one sort of area in which recurrent neural networks are  used to, but also language.  So for example, language is also we construct a sentence and a sentence has a sequential  meaning to it.  And yeah, and the work here is, well, convolutional neural networks not necessarily feedforward  or fully connected neural networks.  So I will explain a little bit about convolutional neural networks as well.  Convolutional neural networks are basically feedforward neural networks with two imposed  constraints.  And basically they substitute the matrix multiplication that was happening here for a convolution  operation.  And I will keep in detail some explanation about the convolution operation.  But here we, in the convolution operation, we have a kernel and this kernel gets cobalt  in our image.  So here we have a picture of our hoaxula.  And we will make a convolution.  I don't know if you guys are familiar with the operation, but it's a, you just put this  matrix over here and you multiply all the elements and just sum them up.  And this gives you one pixel over here.  And then you keep moving forward this sort of operator and you'll get this sort of pixel  values that each time this operator is applied one pixel here.  And at the end, from this handcrafted kernel, I will get some edge detection.  And this edge detection will help us eventually, or could help us eventually to create a system  that is able to recognize something, right?  But the idea of a convolutional neural network is to know or to learn the elements of the  kernel.  So in order to, when it cobalts the image, you get features that are important or that  are more suitable for the classification.  And as I mentioned before, these are the two constraints that are given to a neural network  or to fully connect the neural network in order to convert it into a convolutional neural  network.  One of them is local connectivity, which tells you that not all the input values are going  to be as important for the neural network or that you're only looking at the local patch  of the input values in order to make explicit computation.  So or in more, in our words, a pixel that is located on the far end of the image might  not be as important for a local patch that is located on the other end.  And weight churn, weight churn tells you that a set of pixels or a set of extracted  feature from a local patch of pixels will be important for the other patches.  And therefore, you will assign the same weights to all the patches.  And at the end, you impose these two constraints.  And yeah, magically, your feed-forward neural network is converted into a convolutional  neural network.  And this is one of the images that I like very much because it explicitly tells you how  to calculate the or how the convolutional neural network works.  But in this case, let's think about an RGB image.  An RGB image is a matrix that consists of or three matrices are stuck together.  And we would have a kernel, which is represented by its other tiny matrix, which is also as  a volume of three stack matrices.  And what we do basically is just cobalt the kernel with the input.  So we do this by, for example, looking at this specific x, y in the input feature map.  We will multiply each value d0 times d0 and this one times d minus 1.  And we will do this with the first slice of the kernel.  And then the second slice of the feature map, we will do it with the second slice of the  kernel and so forth for the other ones.  And then once we have computed all these multiplications, all of these will get sum.  And then finally, all these three values that came into scalars will get sum.  And then finally, avias will also get sum.  These summing values at the end are very magically given and also with this something  that was improved later on, which I'll talk about as well.  But yeah, this is the way in which a convolution neural network looks like in the convolution  layer at least.  So it's just you're learning these sort of 3D kernel weights that are cobalt in another  volume of an image, for example.  And another interesting thing is that we have, for example, here two kernels, kernel 1 and  kernel 2.  And kernel 1 gives us another feature map and kernel 2 will give us another feature map.  That feature map is just a modification of the original image by cobalt in the kernel.  So if we have, for example, n or n feature map, sorry, we will have n or sorry, if we  have n kernels, we will have n feature maps at the end.  So we can control the amount of feature maps you want by the amount of kernels that you  have.  And this is one of the most successful convolutional neural networks, which is BG16.  I believe, one of 2014, the ImageNet competition.  ImageNet competition is a competition of classifying 1,000 classes from 1.3 million images.  And this is very incredible result.  I like to think of it as some sort of system that you can integrate into, for example,  into your robot platform and is able to identify 1,000 classes.  So now we could put this into a robot and the robot will walk around and you can take  picture of something and it will say, yeah, this is a computer, yeah, this is a person,  and he will do this for 1,000 classes.  And now we can do this.  And this is very easy to do.  But yeah, these sort of convolutional neural networks are computationally very expensive.  For example, they have around 138 million weights.  But another interesting thing is that, as I explained before, we have these convolutional  layers.  And at the end, you add a fully connected layer, which is a very traditional approach  to construct a neural network.  So you have this convolution, convolution, convolution, convolution, and then you have  the fully connected layers.  And these fully connected layers account for the 90% of all parameters in the network.  So this is also very, very interesting.  Like, it might not be completely convolutional, but it's 90% some fully connected network,  right?  So, yeah.  So another solution to, in order to reduce the amount of parameters that we have in our  fully connected part, is to create or to add a global average pulling layer.  And a global average pulling layer will take the feature maps, and it will just make them  sorry, it will take these feature maps and it will convert them into a single scalar  value.  So in the previous approach, what was happening is that we have the feature maps, and each  feature map gets flattened and then concatenated with all the other ones.  And then from here, we will perform a fully connected layer, and then it will turn out  to be the output values.  But in this case, what we are actually going to perform is we're going to take this feature  map and we want to do a global, we will perform an average of all the elements in the feature  map, and then this average will turn itself into a scalar value.  And then these scalar values will actually be the output nodes.  So why are we performing the average?  This is something that is rather interesting because average gives you information, you  need information of all the single nodes in your convolutional neural network in order  to have something significant.  But if you have something like the maximum value of it, then you're kind of losing information  about it.  So global or the mentality of why you think average is just to give individual representation  to all the nodes in the feature map.  And yes, our initial architecture, which is the one that I proposed in this work, it  consists of nine convolution layers which use global average pulling at the last layer,  and it has around 600,000 parameters.  And it achieves an accuracy of 96 in the IMDb data set and 66 on the third 2013 data set.  This model is 7.5 megabytes, so it's rather small.  And yeah, other methods achieve the same accuracy, but using an ensemble of convolutional neural  networks.  So this also might prove slow for real-time systems.  But this is not the end of the story here because at the end, 600,000 parameters seems  rather cumbersome for a simple task.  So we decided to explore more on modern architectures on convolutional neural networks, and we encountered  this exception architecture, which combines two of the most successful experimental assumptions  in convolutional neural networks that is the use of residual models and depth-wise separable  convolutions.  And the use of these two models or assumptions are basically the state of the R-Rine and  convolutional neural networks.  Which is the residual models that instead of actually performing a sequential operation  on the future maps, we will divide the future map so it adds an identity to it.  And we will perform a convolution again by a nonlinear function, and then we add the  previous future map.  So this is something like a perturbation method in which you have the future map, and you  want to see which perturbations in the future map actually make it better in order for the  classification to be more accurate.  But the most important thing about the residual models is that it makes the back propagation  algorithm which changes the, which is optimization algorithm or which is the algorithm that modifies  the way sufficiently, makes it easy to create a bridge between the previous future maps.  So it's able to propagate the gradient easily through the network, which is this is also  a well-known problem in convolutional neural networks.  That the convolutions get so, or the neural network gets so big that the actual back propagation  of the gradient also gets decreasing or also explodes depending on the uncertain assumptions.  But this allows you to back propagate the gradient easily, and therefore this architecture  one, I believe the 2015 ImageNet, which consists of 101 layers of convolutions, one after each  other.  And the other one, which is that why separable convolutions.  And as I explained before, you have these sort of kernels, right?  These 3D volumes that map the input images with the, that try to convert it to another  set of future maps.  And in this case, the depth-wise separable convolutions try to separate the cross-correlations  between the spatial values and the channel values.  So instead of having 3D volume kernels that will be cobalt in the input feature map, we  will make them channel dependent.  So in this case, we have a channel that has its own kernel.  So all the input feature maps will have a specific kernel, which is, yeah, in this case,  8k times 1, and we will learn to mix the values outputted by the filters in several ways,  which are given by this one-time-one convolution.  So the one-time-one convolution, what it's doing, it's mixing the values given by these  sort of kernels here.  And yeah, this sort of reduction gives you a factor of 1 over n plus 1 over d squared.  And this also eliminated some parameters in our convolutional neural network.  And at the end, our final architecture, which is based on this exception network, was able  to, yeah, we reduced the parameters so 10 times more.  So we have 60,000 parameters now, and the model is around 853 kilobytes.  It achieves almost the same accuracy in the generic classification in the previous model.  If you recall, it's 96.  Here's 95.  And also, emotion classification was exactly the same, which is 66.  And our complete pipeline that included the phase detection and the generic classification,  emotion classification takes around, yeah, I mean, this much of seconds, 0.48 seconds,  right?  And this on a low-end GPU, and yeah, on an I5 CPU was around 0.051 seconds.  So I mean, it's a...  So basically, we reduced the amount of parameters that we actually need in a convolutional neural  network in order to perform a system, a real-time system.  And these are...  I'll show this later.  These are some of our results.  So this is what we want, right?  We have...this is our Robocop team in Italy, and we're performing a competition.  So here we have blue represents women, and the other 100% men.  And yeah, happy men.  I mean, this is very fairly easy to represent, right?  Happy, happy, happy, happy, happy.  This is neutral, if you cannot read it.  And yes, we have also something here, which is SAP men.  But if you...I don't know, it's a...this is...this error here was made for the...because of the  phase detection algorithm, which is the open-CV implementation.  But yeah, if you can look at it more closely, you can see there is some sort of eyebrows  there, and yeah, the map looks at, you know.  And this is another fun thing that I work on, which is...yeah, this is the Solvai conference,  and here is Moudicouli, and then Albert Einstein, and yeah, the other famous physicists, and  it says angry women, which I believe is correct.  And yeah, neutral men.  This movie that I like, which is called 12 Angry Men, and then you can see the other  all of them are angry.  I think none of them are angry.  It's neutral, sad, neutral.  And yeah, I'll show you some of the results that I have in our Github repo.  So, yeah, by the way, this is...these are positive that was...yeah, that I did in order  to put my project and everything, and yeah, this is rather interesting because I published  it without any idea of like how interesting for people could be, but it turned out to  be that a lot of people like it, and yeah, I got a nice contribution from the Opens  of the Community, and here are the same examples.  Well, here's different faces for the emotions, and here's a real sense of me, and I'll explain  later why this might be interesting, because like, okay, this is sad.  I might not look so sad, right?  But this is angry.  I'm not angry here.  This is kind of weird, right?  I'll talk about my errors later on, which is also very interesting.  And yes, perfect.  And yeah, and well, let me just go back here, which is the visualization technique that  I was talking to you about.  This visualization technique is called guided back propagation, and basically what it's  doing, you keep as input the image, and then you wobble the picture, or you can think of  it as wobbling the elements of the picture and seeing which neuron gets activated in  the neural network.  And you look at the neuron that gets activated the most, and then you keep wobbling the pixels,  and you find the pixels that correlate more with the activation of a neuron inside your  convolutional neural network, and therefore it could tell you which pixels in the image  make a neuron activate the most.  And here, for example, I believe every row is an emotion, so apparently this guy was  angry, because yeah, I know Samuel L. Jackson was angry here.  And so we can, for example, see that the neural network responds very heavily on the smile,  obviously, so this is something that, I mean, we expected it from the neural network.  We also see something interesting here, which is that the surprise classification depends  solely on the eyes, or mostly on the eyes.  So how big your eyeballs are open.  So this is also very interesting.  And also, yeah, angry, which is the part that I wanted to talk about.  Like, angry seems that you only have to like really, I don't know how this, you only have  to make your face like this, and you will trick the neural network to believe that you're  actually angry.  So you just have to do this.  Not really, you don't have to be extremely angry.  You just have to really perform this sort of face.  And this is why I was telling you in the real-life model that I didn't look angry.  But yeah, this is why you have also to be careful around these open source implementations,  right?  Because I know what activates the neural network the most, so I can make the faces that activate  it, right?  So you also have to think about it.  And yes, another interesting idea or problem that arose is that most of, as I mentioned  before, yeah, this dataset, right?  I mean, this dataset, it's rather very oriented to Western-looking faces, right?  So I was here in the Hoshul and the Robocop competition what's happening in Japan, and  I received a lot of emails from people in Japan telling me that my implementation didn't  work because it was not working for Asian people.  I mean, it was not working because obviously the dataset is of these people, right?  And also something very interesting that I encountered was the use of glasses.  The neural network confused people that got angry with a person that was only wearing  a glass or wearing glasses like mine, for example, which are very dark.  But also more, most interesting, I think, is that the neural network got conf...  I had a friend of mine and she was sitting in front of it and we had the live demo on  it.  And she didn't have any glasses and it was classifying her as a woman.  She put her glasses on, it said men.  So this is something also very interesting that I would like to pinpoint from the dataset.  I mean, you really have to dig into the dataset to look what errors my... what errors my occur  inside the network, right?  It might be biased to say that because a person is wearing glasses, it might be a man.  Why?  Because all the samples that you give to a neural network of persons wearing glasses  are men, right?  And...  Yeah.  Yeah, so future work.  The reduction of parameters made us create a real-time system, but this is not the end  of how much we can reduce the amount of parameters.  We also can perform some evolutionary strategies in order to reduce the parameters, depending  on maybe the amount of kernels that we have.  This was also not a parameter that I tweaked too much.  So I believe further optimization can be developed here.  Also we can incorporate more classifications.  I mean, the networks that we are turning are so small that we can create more of them in  order to produce, for example, H. And this also H data is in the IMDb dataset.  So we can easily train another neural network for this.  And then the last one, we should create double-headed models, which means that from a single forward  pass, you can output several classes that... several... yeah, several classes that include,  for example, Hapi or that include emotions that include gender and that include H. But  in order to produce this successfully, we will have to have data set that contains the  tree labels.  So that every time we forward pass the image, we have the information of the tree labels.  So this is also something that we do not have and this is something that, therefore, I couldn't  able to make it.  But at least for the H and for the gender that are contained in the same dataset, we can  create this sort of double-headed neural networks.  And yes, thank you for your time.  And I will... how much time do we have?  Is it... we have time?  Yeah, okay.  This is the first... this is sort of half of the presentation that I was trying to give  today.  And if we have time after the questions, I would like to give another talk if you guys  are interested, which is again, convolutional neural networks and offers... yeah, and opens  over a relation to it, which is for image captioning.  But yeah, thank you.  Any questions?  So Chris here read an article about recognizing street signs, just the same sort of neural  networks.  And the question is, how do you know that, okay, we have a stop sign and put some extra  text on it.  And the turning sign and the stopping sign.  And the movement, you know, if it's an octagon and it's red and the stop sign will let the  text on it.  It can be an error or whatever, you know, it's a stop sign.  How can one assure that one of the strings is such a thing that really the most important  things are used for this termination and not something that might work for the image  set but may not work for the real life?  Thank you.  Yeah, this is a very interesting question, which is how can we make sure that our networks  are able to generalize correctly and that the small perturbances in the input data don't  change the output.  And this is a very hot topic right now.  And I believe maybe the paper that you read was on adversarial networks, which they tried  to create different modifications.  So I think there's even some examples in which they add noise, which for us, it's, we cannot  see this noise, one, it's an image.  And then the class changes entirely from what it was assigned, for example, into a panda.  And but the noise is, it's not, we are not able to perceive it.  And this, yeah, this is a well-known problem.  But another thing is that it's well-known for convolutional neural networks, but it's  also well-known for most of the machine learning algorithms.  So it's not only for convolutional neural networks.  So one can also optimize noise that you give to input data to an SVM or something else  that will change the output.  So I mean, this is, this big problem in machine learning in general and not necessarily only  in convolutional neural networks.  I think from the F2P such an example, we want to make sure that it recognizes the box  and that it recognizes the color red.  And only if it's a two or three, it takes it to the stop sign.  So it's much more.  I mean, yeah, the question is that does it really use that black box or does it really  see a train into digital business and then find a rule set how to combine these signals?  Yeah.  I believe what people...  I mean, currently there's, as I mentioned, this is a very hot topic and currently there's  even a competition in Kegel in which they're trying to create adversarial examples so in  order to make the neural networks more robust.  So there's no answer as far as I know right now.  No?  No?  There's this onegravel that's kind of  is your useable role.  What was the mechanism you trained in your network  that would help you pick different kernels and help them  to access rates to the given information of age  or main email information?  And you picked the most successful set of kernels?  Could you tell everyone what was the input set  to secure that for energy?  The training process or the evaluation process?  Yeah, okay, so the question is how is the training  and evaluation process performed?  Well, the training process is, I believe, well known  in neural networks, which is just the implementation  of the back propagation algorithm, which that's why  it's so important because I don't have to choose  the elements in the kernel.  The kernels or the neural network  can learn to choose the values for me.  And the evaluation procedure, well, we basically  divided our dataset into training, validation, and test,  and then we just performed, we used the model  that performed better on the validation,  and then we just proceeded to test it.  Yeah, so the complete, chosen of the values  in the kernel is not my idea and do it.  The optimizer did it for me.  Okay, so what we measure is the end-to-end data set  and what the network is.  Exactly, yeah.  But how do you divide between just trained following  simple tests, over trained, and some under trained  and how do you get the measurement for this?  Is it just the comparement to the end-to-end data  this image success?  Yeah, you have to perform the, this is why you have  the validation set because you want to see that  your validation set is performing correctly, right?  So you split something that you're not training on  and in order to see that it's performing well  and in order that it's able to generalize well.  And yes, this validation set comes only from  the original data set, yes.  In order for back-to-back-propagation to work,  you need to define the error function.  Yes, exactly.  Define for this case because there's multiple things  that you need to account for.  Did you test for a matching protocol for face-to-face  with this, or matching for where it faces the image  and then how it can be used?  So you have a distance measurement for different  features of different specifications.  Yes, the loss function is nothing fancy here.  It's something, you have several loss functions  as you mentioned, you have this square loss, for example.  But here we only use the typical classification  loss entropy loss, which is just tell you the entropy  of how well, or the cross entropy basically.  So it tells you how well your output values  in this looking at them, if there were a probability  distribution, how well it compares to the probability  distribution of the actual thing.  So the cross entropy loss, yes.  So nothing really special.  And we're now just doing research in analyzing the videos  of those images.  Yeah, so the question is whether or not we can incorporate  video analysis in our robots.  And yeah, obviously this is something very interesting as well.  But this is also something that becomes  computationally complicated, right?  Because you have to account for now video,  we'd add another dimension to your neural network.  And, but yeah, there's current research on this,  for example, how you call this, to detect whether  what a person is in actual classification,  yes, actual classification.  Yeah, further questions before I proceed to the next slide?  Yes.  You mentioned the reduction.  So you mentioned that the fully connected layer has  accounts for 90% of all network weights.  But can you name the numbers of the reduction?  So how much weight do you manage to cut in your examples  when you're planning it, one by one,  and the solutions and everything?  How much did you win?  Yeah, this, the question is how much did I win from cutting out  the last fully connected values?  Well, I would have had to train a fully connected network  in order to see how many values are actually,  how many values actually give me the same classification  accuracy.  So I didn't do this.  I proceeded explicitly to perform the global average  pooling, which I knew it was something that already  gave you a reduction.  But as we saw, for example, that incorporation of the exception  architecture, it gave us, yeah, it reduced 10 times more, right?  Yes?  So as a concept, I've seen a mention of, for example,  you want to implement some neural network and it's really  hard to train on your hardware.  So one of the solution is to take, for example,  an Alex Nats, get rid of the fully connected layer,  freeze all the convolutions and everything,  and just train this last fully connected layer to,  let's say, detect only Nats.  So I didn't see this fact about 90%.  So is it the hope to use this sort of approach to train  something easily in home conditions on your GPU?  Have you ever tried something like that?  Yeah, so the question is have I tried doing fine tuning?  Yeah, fine tuning.  Oh, yes, that's the word.  Yes, I've tried doing fine tuning.  But as you mentioned, as you saw in the BE-16 network,  as you saw in the network, the network consists of three  fully connected layers.  So you don't take out the three.  You just take only one.  And then from on top of this one, you train.  Yeah, so you don't get rid of this 90%, right?  Yeah, you still have some.  Cool, yes?  Will the slides be available?  Yes, the slides will be available.  Sorry for that.  Yes.  OK.  Where?  Would you, is this information somewhere already?  There is information of my, of everything here on the,  on my GitHub repository.  So all the information you can find there.  But I'll make sure that also people from Frost can have it.  OK.  Good.  Then I'll proceed to the, I mean, if we have some time,  I guess, we have 20 minutes.  I'll proceed with something that might be also interesting  for people that it's on.  So, yeah.  Yes.  So, what I'm going to present now is image caption and classification  of a normal situation.  This project was also made with Professor Paul Ploeger.  And it's also trained a convolution and an LSTM network  in order to account for situations in which the situation,  in which values might be dangerous.  Or for example, we would like to train a neural network that tells  us that creates a description of the situation.  And this description will give us information on whether or not  the situation should be important for the robot.  So image captioning consists of a convolution neural network  and a LSTM network that in conjunction perform or construct  a sentence of the image.  So the neural network will take the image and it will perform,  it will say a group of young people playing a game of frisbee.  And this is image captioning.  And our basic example or basic thing that we would like to perform  is a system in which a robot sees a situation.  It takes a picture of the situation and then it describes  the situation or it makes a sentence of the situation.  This sort of sentence can be communicated to someone else  and then they could proceed in order to, as they see fit,  regarding whatever is happening.  So for example, in this case, the robot will approach  and instead of having several components in a robot in which  they try to do maybe post-estimation in which they try to do  several classifications, we will have a single component  that will tell us everything that we would like to know  about the situation at hand.  So in this case, the robot will approach, it will take a picture  and then it will say an elderly man is laying unconscious  on the ground.  And this is the system that we created.  And, yeah, LSTMs are another form of recurrent neon network  that are a little bit more complicated.  But, yeah, they are differential versions of the memory chip  in a digital computer.  This is what Alex Reb said, which is a famous researcher  that worked with LSTMs for a long time.  And basically, LSTMs contain several gates that control  the input, the output, and the hidden state of the network.  And I will not go into too much detail about this, but, yes,  the process in which the image captioning system works,  it just passes an image through a pre-trained convolutional  neural network, as I explained before.  It poses this into the first state of the recurrent neural  network, and then the recurrent neural network will start predicting,  give it the information of the image, a word, or the first word  in the sentence.  Then it will use the next value in which it was trained on,  and the hidden state in order to produce the next word in the  sentence.  And it will continue to carry on until it performs a sentence  such as this.  A group of young people playing a game of Street Street  is explicitly given here in which you have a stored token,  you have the information of the image, and then it produces a straw.  And then given the word straw, and the hidden information of the  previous step, it will produce hat.  And then if you give hat and the given information from the  previous two steps, it will give you the token end.  So we'll learn to create a sentence.  And this, there was a problem regarding our data contained in  the, for this case, because there are usually not data sets  that contain anomalous situations.  So we don't have data sets that contain, for example, guns or  people maybe in pain or blot.  So we had to actually go into Flickr, look for these images,  and then ask people around our university to label or to create  a sentence for each of these images.  So this was a very enduring process.  But at the end, we collected 8,000, or sorry, 100, 1,000 and  8 captioned images.  And that were captioned by 20 different people here.  And yeah, all of our images are on the creative common license,  and I hope to make them available soon so people can work on  this also in machine learning.  And these are some of the examples that are recollected in  our data, which include, for example, a person that has been  injured, a police officer holding a gun, and some violent images  and some things running on, being on fire on the street.  And we implemented one of these image captioned models,  and we obtained a score of 14.2, a meter score, which tells  you, which tries to compare sentences, basically, how much  a sentence is in accordance to another reference sentence.  So this sort of evaluation is also very mysterious, because  it's built by humans in which some parameters are also well  tuned, and I believe this metric should improve.  But anyhow, this is the metric that people use in these  captioning models.  And we also obtained an accuracy of 97%, which means that  I'll give you an image, and then I will tell you, I want to  know whether the image is an anomaly or not an anomaly.  So we received an accuracy of 97% for this, and these are  some of our results that we obtained.  So this is the image that we received, and the neural network  said this, man is doing a trick on a skateboard.  Then we gave an image of this, a train traveling down  these tracks next to a lost room field, a group of people  flying cats in the field, pizza with cheese and herbs on the  table, this with a laptop and a monitor.  This is also, I think, very difficult and could also be  a good application for robots.  And then a man on a server riding away in the ocean.  And these are some of the errors that we also got from  these non-anomalies images, which are a woman sitting on a  bed with a laptop computer.  Well, it's not a laptop computer, but it's some sort of  cushion.  And then a dirty toilet in a small room with a wooden floor.  It's rather dirty, but if there's no wooden floor or  anything like that.  But there's wood here.  So I don't know what's happening actually here, right?  So it's also very, we would have to make some sort of  error evaluation of the network to actually see what's  happening there.  A cat sitting on a car seat with a concerned look.  And this work here, concern or concerned look, it's also  very like, why did it happen, right?  Well, I went into the data and I found out that a lot of  people describe images that are looking towards them as  concerned.  So this is why we have the word concerned here, because  it's also biased in the data, right?  And then we discovered that a neuron doesn't know how to  count.  So a cup of coffee and a cup of coffee.  So this might not be entirely wrong, but still it's a man  is eating a sandwich and drinking beer.  And a woman holding wine glasses.  So this is also wrong.  And these are some of the examples that were, that were,  that we captured, that our neural network captioned and  that I believe that they're positive.  So a man is holding a gun.  So it only took the image and was able to create, learn the  English language and then create a sentence out of it.  And then here, house is burning.  A car is crashing on Snow Corbett Street.  There is a woman with blood on the floor.  A man is choking on our man.  A firefighter is trying to put out a fire.  So I mean, this is what we wanted.  We wanted the system that we can incorporate in a robot that  is able to tell us whether some information might be an  anomaly, right?  And these are some of the results in which, yeah, in which  the neural network didn't perform as well for anomalies.  People with red helmets are sitting on a car on A.  So we see that the neural network doesn't always finish the  sentences.  There's a broken window laying on the ground.  I believe that this might be because there's some variation  here on the floor, but I'm not entirely sure.  And a man is showing his injured shoulder.  So it's not showing his shoulder.  And then a man is showing his right arm in which he has a  severe injury.  I mean, this is not his right arm.  This is his leg, but it's difficult, right?  I mean, a man is being held by the police.  So I mean, it's not a police, but referees.  So I mean, a man is dancing with a woman on a dance floor.  I don't know what's happening there, to be honest.  So this is also a difficult caption, right?  Like, what are we asking to the network?  What are we training it for?  And this is also complicated, right?  So we have to look at the images we're giving.  And, yeah, thank you for your time.  This is an honor small project we had.  Thank you.  Questions?  Questions?  Yeah?  Yeah.  Yeah, the question is whether I use, whether I write,  whether I wrote everything from scratch or whether I use a  framework?  Yeah.  I use TensorFlow with Keras, which is the easiest way.  Yeah.  Another question?  Yeah?  Exceptional events data sets pretty small just for thousands.  Yeah, it is rather small, yes.  Aren't you concerned about forbidding more just training  of that particular disease objective test actors?  Yes, no, we perform, yeah, we divided the data set  accordingly like 2020, like 80% for training,  and then from this 80 and our 20 for validation,  and then we did the appropriate values so we don't  overfit it, at least in the classification.  But the captioning part is rather complicated to evaluate.  And this is why I was making a small remark on the metrics  used on the captioning scenario.  It's also very hard coded, so I don't trust it too much.  Yes?  Any other question?  Yeah?  Okay, cool.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_0_47.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_48_167.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_168_323.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_324_15707.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_15708_17915.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_17916_18059.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_18060_20375.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_20376_22595.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_22596_23735.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_23736_24563.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_24564_32411.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_32412_32675.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_32676_32999.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_33000_33179.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_33180_33239.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_33240_33287.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_33288_33971.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_33972_34019.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34020_34139.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34140_34619.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34620_34679.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34680_34763.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34764_34823.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34824_34907.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_34908_37859.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_37860_37907.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_37908_38051.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_38052_38099.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_38100_38219.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_38220_43403.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_43404_53459.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_53460_53507.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_53508_54815.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_54816_55127.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_55128_55295.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_55296_55439.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_55440_55991.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_55992_56051.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_56052_56099.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_56100_56315.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_56316_56591.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_56592_56639.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_56640_58547.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_58548_59855.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_59856_60131.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_60132_60863.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_60864_60935.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_60936_61055.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61056_61115.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61116_61175.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61176_61271.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61272_61319.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61320_61415.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61416_61895.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_61896_66083.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_66084_66719.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_66720_70151.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_70152_70283.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_70284_71435.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_71436_72755.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_72756_73463.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_73464_73955.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_73956_74519.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_74520_75467.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_75468_75527.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_75528_75659.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_75660_78275.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_78276_78347.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_78348_78395.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_78396_78503.png froscon2017-2037-real-time_face_detection_and_emotion_gender_classification_with_convolutional_neura_78504_78517.png