Snorkel Beambell - Real-time Weak Supervision on Apache Flink
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Marthi, Suneel
The advent of Deep Learning models has led to a massive growth of real-world machine learning. Deep Learning allows Machine Learning Practitioners to get the state-of-the-art score on benchmarks without any hand-engineered features. These Deep Learning models rely on massive hand-labeled training datasets which is a bottleneck in developing and modifying machine learning models. Most large scale Machine Learning systems today like Google’s DryBell use some form of Weak Supervision to construct lower quality, large scale training datasets that can be used to continuously retrain and deploy models in a real-world scenario. The challenge with continuous retraining is that one needs to maintain prior state (e.g., the learning functions in case of Weak Supervision or a pre-trained model like BERT or Word2Vec for Transfer Learning) that is shared across multiple streams, while continuously updating the model. Apache Beam’s Stateful Stream processing capabilities are a perfect match here including support for scalable Weak Supervision.
 Alright, next up we have Sunil talking about machine learning with Beam and Apache Flink. Hi everyone, this is my second time at Foster, second consecutive year. So folks who have been in the, okay. I didn't ask for that, that's fine. So folks who have been here last year, maybe I've seen our talk about neural machine translation and I had a live demo of translating German tweets about foster on stage. But yeah, this time around there'll be no demo for what I'm presenting about. So today we'll be talking about something called weak supervision and how many folks here familiar with that weak supervision or weak learning? There you go. Okay. Okay. So just I'm a member of Apache Foundation and I do contribute code to a bunch of projects including Beam and Flink and the ones that you see there on the screen. My day job, I work for AWS. So the agenda for today is what is weak supervision? Since I didn't see many hands go up, let's talk about what is weak supervision and why do we need it? And what kind of problems are we trying to solve with weak supervision and when do we need to use it? And so when you talk about weak supervision, there's something called learning functions and we'll be looking at what learning functions are. And once you do all of that, how do you scale it out? Obviously, you need to scale it out on a distributed platform. So if folks have heard about Google's paper last year, it's called Google Snorkel Dry Bell. And so basically they're trying to scale out weak learning on Google's internal version of Hadoop MapReduce. But we'll be trying to scale this out on Apache Beam. And we'll be looking at sample use cases. So the OSS tools that we'll be talking about as part of the stock is a framework called Snorkel from Stanford University. And it's a framework for programmatically trying to build and manage training data. And we'll see how it actually does that. And of course, we're going to run all of that on Beam and Flink as a back-end runner. So I guess folks are familiar with Flink and Beam here. Two, three, four, five, six, that's it. And well, let me not talk about stateful computations in Flink then and how stateful computations can be leveraged for real-time machine learning training. So what is the biggest challenge that we have when we talk about machine learning, especially deep learning training? What's the biggest challenge that folks who have raised their hands? What's that? Yeah, trying to get labeled training data. And especially if you're training deep learning models with images of cats and dogs, you need a million images of cats and dogs and labeled as cat and a dog, which is very hard to get. So the difference between weak supervision and the most more common supervised learning is that in common supervised learning, you have a bunch of training data which has been labeled. So set of data points X1, X2, X3. And then you have a bunch of K classes. You're trying to classify this into one of K classes, assign a label to one of the K classes. And then use a classification algorithm like a night base or logistic regression to actually classify the X data point into one of the Y classes. And as part of any supervised learning, what we do is what we have for supervised learning is we have a set of ground truth labels. So somebody has human as annotated, this data point is class A1, class 2, class 3. So you have the ground truth labels. And this is usually done by a subject matter experts. So for example, if I'm working with satellite images, right? So I'm trying to train a model for predicting like tulip fields. If you have seen my earlier talk in Berlin buzzwords about trying to predict tulip fields from satellite images, a human has annotated, has gone and drawn a bounding box as to whether tulip beds are on my training data, which is not possible given the amount of satellite images that get downloaded every week. So the traditional approach is that you have a human label that training data for you. And then you try to minimize the loss function. You try to train a class classifier and minimize the loss function like most common is using a stochastic gradient descent or it could be a cross entropy. But what if you don't have the training data and you don't have the ground truth? And let's say if I'm at Google and the biggest bottleneck that we see is that managing training data sets by hand is one of the biggest bottlenecks in machine learning today. So if you're dealing with Google scale or Amazon scale, you just cannot human cannot annotate each and every billion data points that are needed for training classifiers. And what is the other option? Well, why did you just crowdsource it put it on mechanical Turk and let humans do it? Sure, but you need specialist knowledge, right? So if I'm trying to annotate like a radiology images or X-rays or mammograms, I cannot just crowdsource it to Tom Dick and Harry and college grad students don't get paid in peer doctors don't get paid in pizzas or t-shirts. They need real money to actually for them to do that being subject matter experts. So the point here is that as we scale and the creation of label training data becomes increasingly expensive because the lack of subject matter expertise, which is impossible to get in most cases. So and this eventually leads to not being able to kickstart your machine learning training models. So unlike the big internet companies, if I'm a startup, I don't have much data to start up with. So what do I do? And so that's one of the challenges for a startup, right? And it's going to take me a few years for me to build up my data sets and at which point I can say, okay, I have enough data to train a machine learning model. But again, I need a human to go back and annotate all the data and label the data. And moreover, since deep learning is the fad and everything is deep learning. And if you say that you're doing AI and you're doing and you have said, okay, and you don't claim that you have done deep learning. Yeah, it's not, it's not going to work, right? And deep learning obviously needs massive amounts of data, trained label data. Okay, so with that, obviously the lack of subject matter expertise. And if I'm trying to annotate satellite images and you know, trying to identify the landforms in satellite images, I need to be a geologist or you know, someone specialized in satellite photogrammetry to be able to do that. Similarly for radiology images, you need to be a doctor or a radiologist who can annotate those images, right? And obviously doing it by hand is one of the biggest bottlenecks. So what are the possible solutions that we can look at? So this is a chart that I found on the internet. And so basically the, how do you get label training data? And what are the different ways of doing it? Right? So the very first one talks about traditional supervision. Traditional supervision is you have a human who manually annotates your data and labels your data, right? And the downside to that is it's too expensive and too time consuming. So they came up with a different alternative. So it's called active learning. What is active learning? Active learning is you take a subset of your data that you believe is important for training your machine learning algorithm and then have a human annotate only that particular subset of data. And you can feed that data. If you look at the arrow going that way, you can feed that subset of data into a weak supervision approach. So the next step talks about semi-supervised learning. Semi-supervised learning is that you have a lot of unlabeled data and then you would randomly go and let's say run a clustering algorithm on your data. It's unsupervised learning. And then you look at your clusters and you see how the data points fall in clusters. Then you assign labels to the clusters. You go back and you know, change, you assign those labels to the clusters. You go back, run a clustering algorithm again with a different set of parameters, see what the outcome is. So that's kind of a semi-supervised approach. The third approach is what we are talking about today, weak supervision. So in weak supervision, what we do is we get a lower quality, a low quality, inaccurate quality and very imprecise set of label training data. And this is done by a subject matter expert. So let's say if I have billion data points, I take about like 10,000 data points sample and I have a human annotate that. Then I take that 10,000 sample and then I go and train a classifier to go and label the rest of the data, training data for me. And we'll be looking at that. So how do we do that? So going down the chart, so basically you get first to get cheaper lower quality labels from non-experts and then you could use something like a distant learning. Like you could have a similar use case. So if I'm trying to, if I'm doing a text classification, I can use one of YouTube's classifier algorithms models to actually take that algorithm to classify my text, label my text. And then basically it's going to create a very noisy and very inaccurate level of labels. And then I think it also come up with heuristics. So if the comments, if the text has something like this movie sucks, you know that this, it's a negative sentiment. And if somebody's always keeps saying that it's great movie, great movie, and it's great movie, see it five stars, you know, it's a positive sentiment, right? You can come up with bunch of heuristics, which are called learning functions. And then you can run this learning functions and a distance supervision and some human annotated data, all of that together, you build models individually and aggregate those models to train a classifier to just to label the rest of your data. Okay, so obviously folks have heard about transfer learning. So transfer learning is you have a pre-trained model, the model that has been pre-trained already on a different data set. And then what you want to do is you want to take that model and train your data on the parameters of that model, right? And then use that as a classifier. So moving forward, let's move on to the topic of today. What is week supervision? So week supervision is basically about leveraging higher level, noisier input data from subject matter experts. So what does that mean? I have a sample of my data and I have the data annotated by subject matter experts and I use that sample to train a classifier to label the rest of my universe, training universe. So how is this different from the supervised learning setting that we saw earlier? So the difference is that we don't have number one, we don't have a ground truth. Like the supervised setting, we don't have a ground truth. And we have a bunch of unlabeled data, X number of points. So for each of those points, we assign a score. Score is from 1 to M and this score is assigned by a human subject matter expert. And basically what the human does is he takes a sample of the data and basically the sample of data is the data that he's familiar with, the distribution that he's familiar with. He annotates that. And since it's a kind of a bias built into that, it's not very accurate. The outcome is not very accurate, but that's something that we need to address down the line. So the labels that are assigned in the previous step are called weak labels. And what is the intent of weak labels? Number one, the intent is to decrease the cost and increase the efficiency of getting human annotated data. So instead of training, instead of labeling all my billion data points, if I'm a Google or Amazon, I just take a sample of 1,000 or 10,000 and I have a subject matter expert annotate that. And yes, you're doing that because you're leveraging a subject matter expert and it gets less precise. And you take this label data as well as you create some heuristics and the heuristics are again created by the subject matter expert. So we look at an example of what heuristics would look like in the next step. But when we talk about weak labels, there are two types of weak labels. Number one, the labels are very unprecise. So if I'm trying to classify my data as spam or ham or I don't want to give any label to it, I can abstain from giving a label. So basically if I'm a subject matter expert and I look at the text and I think, okay, this looks like it's a negative sentiment. So I give it a negative label and then this could be positive. And this one kind of seems neutral. So I just don't give it a label at all. I just leave it unlabeled. So for example, in a spam classification, I can have a rule that says return spam if you see contact us in the text. So in a YouTube comment, why would you have a contact us? So just leveraging that knowledge, you can create some labeling functions. And in most cases, what they also do is that you can leverage a third party model. Like for example, you can leverage a model that's been trained on cats and dogs and you know, annotate. And you can use that for annotating images about cheetahs and tigers. And so that's a third party model. So this, how do you generate labeled data? Okay, so if you look at this at the very left, you have a domain subject matter expert. And he's got a bunch of sources that already predefined sources, labels from a different data source. And he's also annotating some data, labeling some data. And he's also creating something called learning functions. So you have three different data sources coming in and three different means of getting data sources. And then for each of the data points, you assign some probability values less than one. And then you got to aggregate that. How do you aggregate all of these models together? And moreover, remember that these, the data that you're getting here is very noisy, inaccurate, and not reliable, right? It's kind of a week. That's the reason why it's called a week supervision. And you got to, you got to minimize the noise and increase the accuracy as before you aggregate that. So I get the data sources, then I train a classifier to look at all the data sources that I have, the weak labels. And then I try to reduce the noise and increase the accuracy. And I use that to go and label the rest of my training data. So which brings us to the topic, a framework called Snorkel from Stanford University, which does all of the stuff that I just described. So basically Snorkel is an open source project from Stanford University. Those are the guys who are working on week supervision. And it's for programmatically trying to build and manage your training data sets. So instead of spending hours and days and trying to get a human to annotate every data point, you can just kind of have it done programmatically in weeks or even hours. So Google has been working with Snorkel on this. And if you have seen this tweet from last year from Jeff Dean, which I'll just let you guys read it. Okay. So basically hand labeling training data is very expensive. And so hence even Google has collaborated with Stanford on the Snorkel framework. And Google has used Snorkel to scale it out for labeling massive data sets at Google using and I think they called the paper. There was a paper called Snorkel Drivebell last year. So when I saw the paper, I was like, okay, so Drivebell is basically Google's version of map reduce, internal map reduce for scaling out week supervision. So I was like, okay, you have Apache Beam. So if you have to do it in open source for the rest of the world, so why not use the same Apache Beam and call it Snorkel Beambell? So that's where the, it's an update on Drivebell. So before we go there, let's talk about labeling functions. So we talked about annotating data and, sorry, we talked about annotating data and getting some kind of heuristic rules. So let's look at some heuristic rules. And these heuristic rules are called labeling functions. So the idea of labeling functions is that initially my data is not labeled here and I have blanks. And then I use my labeling functions to assign some values to the labels. It could be x or a zero. So for example, if I'm doing a spam classification and one of the rules, heuristic rules could be that look for a keyword. So many spam comments talk about my channel, my video, etc. So you call it a spam if you see my in the text, anywhere in the comments text. So that's one rule. Basically here you're doing a keyword match. And the next one could be like doing a reggit's match, regular expression match. So, so nobody would say spam in a spam comments like if I have a YouTube video, nobody would say, okay, check out my video as part of the comments on my video. So if I see check out something like that, if I look for a regular expression, I know that this is a spam. And the other one is just by using heuristics. If the length of the text data is less than some, less than some x length, right, then you say it's not spam. If it is spam, usually, you know, people write a very long text. Otherwise, you cannot sure about it just abstain. Don't give it any label. So the fourth option is that you can use a third party model to label your data. So now that we have seen four different ways of generating labeling functions, heuristics, reggit's, keyword match and a third party model. The next step is you kind of apply all of that. You apply all the four functions to each data point and you have a matrix here. So the columns are the different labeling functions and the rows are the data points. And for each data point, I get a score of one if it is one, if it's not, if it's not spam, zero, if it is a spam and minus one, if I just don't give it a label. So you see that it's for X one for reggit's and heuristic match. There's an overlap. So it says it's spam, right. And for X two, it says it's spam, but reggit's matches. It's it just did not give it a label. And here the keyword matches. It's a spam, but you know, the others just did not give it a label. So these are the inaccuracies you got to deal with. So how do you handle this inaccuracies? So for this, the next step is you take this matrix and you train a label model. You have something called label model that's part of the snorkel framework that kind of tries to minimize the accuracies and increase the reliability of your output. So once you have that data, once you have your training data and your training data as the labels assigned, you can train a real classifier, you can train a classifier to go and label the rest of your data set. And now that data set can then be used for training your deep learning models in pie torture, Keras. So that's so so far we talked about the big supervision part. So I've trained my model, I've labeled my data and I have my data to train a Keras model. Now how do I scale this? I can run it in a single threaded mode or I can scale it across the cluster. So most of this stuff is in Python. And obviously how do you scale? You want to run it in a batch or do you want to run it as a streaming process? Okay. I'll just let you guys read that. And sure, when you talk about batch or stream, the next question is Spark or Flink. I'll let you guys read that. Okay. So since most of the data is in Python and most of it is Python, let's just go with Beam. Beam supports Python API. And it supports, it has a Python Java and Go SDKs and you can support Flink, Spark and Google data flow as runners under the hood. So I'll skip the slide. So the workflow is for the training part, you have a set of unlabeled datasets and all of this is running as a Beam pipeline. You have unlabeled datasets and for unlabeled datasets, you define a set of labeling functions that we talked about and you apply those labeling functions on this unlabeled dataset. It's called P collection in Beam terms. And then once you have the P collection labeled, you do a group by and then you run that by the model. You create your matrix and run that by the model to make it more accurate. And then you output the training data. And this output training data can be used for training your Kerrers or PyTorch models. And this whole pipeline is running on a Beam with the Flink backend. Yeah. So this slide shows you the complete pipeline and the links, questions. Sorry. Thank you, Sunil. So any questions and please stay seated while we're having the questions. Practical example I have is like satellite images, right? Let's say if I'm trying to identify the rivers and the farms from satellite images, I need to, I need my bonding boxes, right? I need my bonding boxes assigned and that needs to be done by a human. And also if you want to determine whether the river is in a city or if it's in the rural area, you need to kind of find the distance between the bonding boxes too. So that's, yeah, and I'm downloading millions of images. I cannot annotate all of that. So what I do is I take a sample and annotate that and then I do some kind of a data augmentation which also can be done. So just so that you can increase your training dataset and I can use those bonding boxes to come up with spatial features and spatial features are my learning functions now, the spatial features, the distance between the bonding boxes and I can go and annotate the rest of my image data. That's one example. And of course if you're doing NLP text classification, you'll have plenty of other examples too. Yeah, thanks for the talk. I had the question you were showing this matrix where you had the output from the different learning functions. Yeah. And I didn't completely understand how you, okay. So the learning functions are a bunch of heuristics that you define and you got to apply those heuristics on each data point in your unlabeled dataset and the learning functions, they come with a score, whether it's a spam or a ham or they just don't vote at all. It's just as abstract. So you collect all the output of all the learning functions as a matrix. So the each row in the matrix is each data point and the columns are the different learning functions with their outputs and the cells are the outputs. And then you take that and train a classifier to, and these learning functions are not accurate. Okay. They're very inaccurate, very imprecise. So you're trying to reduce the noise and increase the accuracy by next training the classifier with that. Okay. Thanks. Yeah. Any more questions? Going once, going twice. Okay. Thank you very much. Thank you very much, San Diego.
 Alright, next up we have Sunil talking about machine learning with Beam and Apache Flink.  Hi everyone, this is my second time at Foster, second consecutive year.  So folks who have been in the, okay.  I didn't ask for that, that's fine.  So folks who have been here last year, maybe I've seen our talk about neural machine translation  and I had a live demo of translating German tweets about foster on stage.  But yeah, this time around there'll be no demo for what I'm presenting about.  So today we'll be talking about something called weak supervision and how many folks  here familiar with that weak supervision or weak learning?  There you go.  Okay.  Okay.  So just I'm a member of Apache Foundation and I do contribute code to a bunch of projects  including Beam and Flink and the ones that you see there on the screen.  My day job, I work for AWS.  So the agenda for today is what is weak supervision?  Since I didn't see many hands go up, let's talk about what is weak supervision and why  do we need it?  And what kind of problems are we trying to solve with weak supervision and when do we  need to use it?  And so when you talk about weak supervision, there's something called learning functions  and we'll be looking at what learning functions are.  And once you do all of that, how do you scale it out?  Obviously, you need to scale it out on a distributed platform.  So if folks have heard about Google's paper last year, it's called Google Snorkel Dry  Bell.  And so basically they're trying to scale out weak learning on Google's internal version  of Hadoop MapReduce.  But we'll be trying to scale this out on Apache Beam.  And we'll be looking at sample use cases.  So the OSS tools that we'll be talking about as part of the stock is a framework called  Snorkel from Stanford University.  And it's a framework for programmatically trying to build and manage training data.  And we'll see how it actually does that.  And of course, we're going to run all of that on Beam and Flink as a back-end runner.  So I guess folks are familiar with Flink and Beam here.  Two, three, four, five, six, that's it.  And well, let me not talk about stateful computations in Flink then and how stateful computations  can be leveraged for real-time machine learning training.  So what is the biggest challenge that we have when we talk about machine learning, especially  deep learning training?  What's the biggest challenge that folks who have raised their hands?  What's that?  Yeah, trying to get labeled training data.  And especially if you're training deep learning models with images of cats and dogs, you need  a million images of cats and dogs and labeled as cat and a dog, which is very hard to get.  So the difference between weak supervision and the most more common supervised learning  is that in common supervised learning, you have a bunch of training data which has been  labeled.  So set of data points X1, X2, X3.  And then you have a bunch of K classes.  You're trying to classify this into one of K classes, assign a label to one of the K classes.  And then use a classification algorithm like a night base or logistic regression to actually  classify the X data point into one of the Y classes.  And as part of any supervised learning, what we do is what we have for supervised learning  is we have a set of ground truth labels.  So somebody has human as annotated, this data point is class A1, class 2, class 3.  So you have the ground truth labels.  And this is usually done by a subject matter experts.  So for example, if I'm working with satellite images, right?  So I'm trying to train a model for predicting like tulip fields.  If you have seen my earlier talk in Berlin buzzwords about trying to predict tulip fields  from satellite images, a human has annotated, has gone and drawn a bounding box as to whether  tulip beds are on my training data, which is not possible given the amount of satellite  images that get downloaded every week.  So the traditional approach is that you have a human label that training data for you.  And then you try to minimize the loss function.  You try to train a class classifier and minimize the loss function like most common is using  a stochastic gradient descent or it could be a cross entropy.  But what if you don't have the training data and you don't have the ground truth?  And let's say if I'm at Google and the biggest bottleneck that we see is that managing training  data sets by hand is one of the biggest bottlenecks in machine learning today.  So if you're dealing with Google scale or Amazon scale, you just cannot human cannot  annotate each and every billion data points that are needed for training classifiers.  And what is the other option?  Well, why did you just crowdsource it put it on mechanical Turk and let humans do it?  Sure, but you need specialist knowledge, right?  So if I'm trying to annotate like a radiology images or X-rays or mammograms, I cannot just  crowdsource it to Tom Dick and Harry and college grad students don't get paid in peer doctors  don't get paid in pizzas or t-shirts.  They need real money to actually for them to do that being subject matter experts.  So the point here is that as we scale and the creation of label training data becomes  increasingly expensive because the lack of subject matter expertise, which is impossible  to get in most cases.  So and this eventually leads to not being able to kickstart your machine learning training  models.  So unlike the big internet companies, if I'm a startup, I don't have much data to start  up with.  So what do I do?  And so that's one of the challenges for a startup, right?  And it's going to take me a few years for me to build up my data sets and at which point  I can say, okay, I have enough data to train a machine learning model.  But again, I need a human to go back and annotate all the data and label the data.  And moreover, since deep learning is the fad and everything is deep learning.  And if you say that you're doing AI and you're doing and you have said, okay, and you don't  claim that you have done deep learning.  Yeah, it's not, it's not going to work, right?  And deep learning obviously needs massive amounts of data, trained label data.  Okay, so with that, obviously the lack of subject matter expertise.  And if I'm trying to annotate satellite images and you know, trying to identify the landforms  in satellite images, I need to be a geologist or you know, someone specialized in satellite  photogrammetry to be able to do that.  Similarly for radiology images, you need to be a doctor or a radiologist who can annotate  those images, right?  And obviously doing it by hand is one of the biggest bottlenecks.  So what are the possible solutions that we can look at?  So this is a chart that I found on the internet.  And so basically the, how do you get label training data?  And what are the different ways of doing it?  Right?  So the very first one talks about traditional supervision.  Traditional supervision is you have a human who manually annotates your data and labels  your data, right?  And the downside to that is it's too expensive and too time consuming.  So they came up with a different alternative.  So it's called active learning.  What is active learning?  Active learning is you take a subset of your data that you believe is important for training  your machine learning algorithm and then have a human annotate only that particular subset  of data.  And you can feed that data.  If you look at the arrow going that way, you can feed that subset of data into a weak  supervision approach.  So the next step talks about semi-supervised learning.  Semi-supervised learning is that you have a lot of unlabeled data and then you would  randomly go and let's say run a clustering algorithm on your data.  It's unsupervised learning.  And then you look at your clusters and you see how the data points fall in clusters.  Then you assign labels to the clusters.  You go back and you know, change, you assign those labels to the clusters.  You go back, run a clustering algorithm again with a different set of parameters, see what  the outcome is.  So that's kind of a semi-supervised approach.  The third approach is what we are talking about today, weak supervision.  So in weak supervision, what we do is we get a lower quality, a low quality, inaccurate  quality and very imprecise set of label training data.  And this is done by a subject matter expert.  So let's say if I have billion data points, I take about like 10,000 data points sample  and I have a human annotate that.  Then I take that 10,000 sample and then I go and train a classifier to go and label  the rest of the data, training data for me.  And we'll be looking at that.  So how do we do that?  So going down the chart, so basically you get first to get cheaper lower quality labels  from non-experts and then you could use something like a distant learning.  Like you could have a similar use case.  So if I'm trying to, if I'm doing a text classification, I can use one of YouTube's classifier algorithms  models to actually take that algorithm to classify my text, label my text.  And then basically it's going to create a very noisy and very inaccurate level of labels.  And then I think it also come up with heuristics.  So if the comments, if the text has something like this movie sucks, you know that this,  it's a negative sentiment.  And if somebody's always keeps saying that it's great movie, great movie, and it's great  movie, see it five stars, you know, it's a positive sentiment, right?  You can come up with bunch of heuristics, which are called learning functions.  And then you can run this learning functions and a distance supervision and some human annotated  data, all of that together, you build models individually and aggregate those models to  train a classifier to just to label the rest of your data.  Okay, so obviously folks have heard about transfer learning.  So transfer learning is you have a pre-trained model, the model that has been pre-trained  already on a different data set.  And then what you want to do is you want to take that model and train your data on the  parameters of that model, right?  And then use that as a classifier.  So moving forward, let's move on to the topic of today.  What is week supervision?  So week supervision is basically about leveraging higher level, noisier input data from subject  matter experts.  So what does that mean?  I have a sample of my data and I have the data annotated by subject matter experts and  I use that sample to train a classifier to label the rest of my universe, training universe.  So how is this different from the supervised learning setting that we saw earlier?  So the difference is that we don't have number one, we don't have a ground truth.  Like the supervised setting, we don't have a ground truth.  And we have a bunch of unlabeled data, X number of points.  So for each of those points, we assign a score.  Score is from 1 to M and this score is assigned by a human subject matter expert.  And basically what the human does is he takes a sample of the data and basically the sample  of data is the data that he's familiar with, the distribution that he's familiar with.  He annotates that.  And since it's a kind of a bias built into that, it's not very accurate.  The outcome is not very accurate, but that's something that we need to address down the  line.  So the labels that are assigned in the previous step are called weak labels.  And what is the intent of weak labels?  Number one, the intent is to decrease the cost and increase the efficiency of getting  human annotated data.  So instead of training, instead of labeling all my billion data points, if I'm a Google  or Amazon, I just take a sample of 1,000 or 10,000 and I have a subject matter expert  annotate that.  And yes, you're doing that because you're leveraging a subject matter expert and it gets less precise.  And you take this label data as well as you create some heuristics and the heuristics  are again created by the subject matter expert.  So we look at an example of what heuristics would look like in the next step.  But when we talk about weak labels, there are two types of weak labels.  Number one, the labels are very unprecise.  So if I'm trying to classify my data as spam or ham or I don't want to give any label to  it, I can abstain from giving a label.  So basically if I'm a subject matter expert and I look at the text and I think, okay,  this looks like it's a negative sentiment.  So I give it a negative label and then this could be positive.  And this one kind of seems neutral.  So I just don't give it a label at all.  I just leave it unlabeled.  So for example, in a spam classification, I can have a rule that says return spam if  you see contact us in the text.  So in a YouTube comment, why would you have a contact us?  So just leveraging that knowledge, you can create some labeling functions.  And in most cases, what they also do is that you can leverage a third party model.  Like for example, you can leverage a model that's been trained on cats and dogs and you  know, annotate.  And you can use that for annotating images about cheetahs and tigers.  And so that's a third party model.  So this, how do you generate labeled data?  Okay, so if you look at this at the very left, you have a domain subject matter expert.  And he's got a bunch of sources that already predefined sources, labels from a different  data source.  And he's also annotating some data, labeling some data.  And he's also creating something called learning functions.  So you have three different data sources coming in and three different means of getting data  sources.  And then for each of the data points, you assign some probability values less than one.  And then you got to aggregate that.  How do you aggregate all of these models together?  And moreover, remember that these, the data that you're getting here is very noisy, inaccurate,  and not reliable, right?  It's kind of a week.  That's the reason why it's called a week supervision.  And you got to, you got to minimize the noise and increase the accuracy as before you aggregate  that.  So I get the data sources, then I train a classifier to look at all the data sources  that I have, the weak labels.  And then I try to reduce the noise and increase the accuracy.  And I use that to go and label the rest of my training data.  So which brings us to the topic, a framework called Snorkel from Stanford University, which  does all of the stuff that I just described.  So basically Snorkel is an open source project from Stanford University.  Those are the guys who are working on week supervision.  And it's for programmatically trying to build and manage your training data sets.  So instead of spending hours and days and trying to get a human to annotate every data  point, you can just kind of have it done programmatically in weeks or even hours.  So Google has been working with Snorkel on this.  And if you have seen this tweet from last year from Jeff Dean, which I'll just let you  guys read it.  Okay.  So basically hand labeling training data is very expensive.  And so hence even Google has collaborated with Stanford on the Snorkel framework.  And Google has used Snorkel to scale it out for labeling massive data sets at Google using  and I think they called the paper.  There was a paper called Snorkel Drivebell last year.  So when I saw the paper, I was like, okay, so Drivebell is basically Google's version  of map reduce, internal map reduce for scaling out week supervision.  So I was like, okay, you have Apache Beam.  So if you have to do it in open source for the rest of the world, so why not use the  same Apache Beam and call it Snorkel Beambell?  So that's where the, it's an update on Drivebell.  So before we go there, let's talk about labeling functions.  So we talked about annotating data and, sorry, we talked about annotating data and getting  some kind of heuristic rules.  So let's look at some heuristic rules.  And these heuristic rules are called labeling functions.  So the idea of labeling functions is that initially my data is not labeled here and I have blanks.  And then I use my labeling functions to assign some values to the labels.  It could be x or a zero.  So for example, if I'm doing a spam classification and one of the rules, heuristic rules could  be that look for a keyword.  So many spam comments talk about my channel, my video, etc.  So you call it a spam if you see my in the text, anywhere in the comments text.  So that's one rule.  Basically here you're doing a keyword match.  And the next one could be like doing a reggit's match, regular expression match.  So, so nobody would say spam in a spam comments like if I have a YouTube video, nobody would  say, okay, check out my video as part of the comments on my video.  So if I see check out something like that, if I look for a regular expression, I know  that this is a spam.  And the other one is just by using heuristics.  If the length of the text data is less than some, less than some x length, right, then  you say it's not spam.  If it is spam, usually, you know, people write a very long text.  Otherwise, you cannot sure about it just abstain.  Don't give it any label.  So the fourth option is that you can use a third party model to label your data.  So now that we have seen four different ways of generating labeling functions, heuristics,  reggit's, keyword match and a third party model.  The next step is you kind of apply all of that.  You apply all the four functions to each data point and you have a matrix here.  So the columns are the different labeling functions and the rows are the data points.  And for each data point, I get a score of one if it is one, if it's not, if it's not  spam, zero, if it is a spam and minus one, if I just don't give it a label.  So you see that it's for X one for reggit's and heuristic match.  There's an overlap.  So it says it's spam, right.  And for X two, it says it's spam, but reggit's matches.  It's it just did not give it a label.  And here the keyword matches.  It's a spam, but you know, the others just did not give it a label.  So these are the inaccuracies you got to deal with.  So how do you handle this inaccuracies?  So for this, the next step is you take this matrix and you train a label model.  You have something called label model that's part of the snorkel framework that kind of  tries to minimize the accuracies and increase the reliability of your output.  So once you have that data, once you have your training data and your training data  as the labels assigned, you can train a real classifier, you can train a classifier to go  and label the rest of your data set.  And now that data set can then be used for training your deep learning models in pie  torture, Keras.  So that's so so far we talked about the big supervision part.  So I've trained my model, I've labeled my data and I have my data to train a Keras model.  Now how do I scale this?  I can run it in a single threaded mode or I can scale it across the cluster.  So most of this stuff is in Python.  And obviously how do you scale?  You want to run it in a batch or do you want to run it as a streaming process?  Okay.  I'll just let you guys read that.  And sure, when you talk about batch or stream, the next question is Spark or Flink.  I'll let you guys read that.  Okay.  So since most of the data is in Python and most of it is Python, let's just go with Beam.  Beam supports Python API.  And it supports, it has a Python Java and Go SDKs and you can support Flink, Spark and  Google data flow as runners under the hood.  So I'll skip the slide.  So the workflow is for the training part, you have a set of unlabeled datasets and all  of this is running as a Beam pipeline.  You have unlabeled datasets and for unlabeled datasets, you define a set of labeling functions  that we talked about and you apply those labeling functions on this unlabeled dataset.  It's called P collection in Beam terms.  And then once you have the P collection labeled, you do a group by and then you run that by  the model.  You create your matrix and run that by the model to make it more accurate.  And then you output the training data.  And this output training data can be used for training your Kerrers or PyTorch models.  And this whole pipeline is running on a Beam with the Flink backend.  Yeah.  So this slide shows you the complete pipeline and the links, questions.  Sorry.  Thank you, Sunil.  So any questions and please stay seated while we're having the questions.  Practical example I have is like satellite images, right?  Let's say if I'm trying to identify the rivers and the farms from satellite images, I need  to, I need my bonding boxes, right?  I need my bonding boxes assigned and that needs to be done by a human.  And also if you want to determine whether the river is in a city or if it's in the rural  area, you need to kind of find the distance between the bonding boxes too.  So that's, yeah, and I'm downloading millions of images.  I cannot annotate all of that.  So what I do is I take a sample and annotate that and then I do some kind of a data augmentation  which also can be done.  So just so that you can increase your training dataset and I can use those bonding boxes  to come up with spatial features and spatial features are my learning functions now, the  spatial features, the distance between the bonding boxes and I can go and annotate the  rest of my image data.  That's one example.  And of course if you're doing NLP text classification, you'll have plenty of other examples too.  Yeah, thanks for the talk.  I had the question you were showing this matrix where you had the output from the different  learning functions.  Yeah.  And I didn't completely understand how you, okay.  So the learning functions are a bunch of heuristics that you define and you got to apply those  heuristics on each data point in your unlabeled dataset and the learning functions, they come  with a score, whether it's a spam or a ham or they just don't vote at all.  It's just as abstract.  So you collect all the output of all the learning functions as a matrix.  So the each row in the matrix is each data point and the columns are the different learning  functions with their outputs and the cells are the outputs.  And then you take that and train a classifier to, and these learning functions are not accurate.  Okay.  They're very inaccurate, very imprecise.  So you're trying to reduce the noise and increase the accuracy by next training the  classifier with that.  Okay.  Thanks.  Yeah.  Any more questions?  Going once, going twice.  Okay.  Thank you very much.  Thank you very much, San Diego.
snorkel_beambell_0_167.png snorkel_beambell_168_1271.png snorkel_beambell_1272_2687.png snorkel_beambell_2688_5243.png snorkel_beambell_5244_8375.png snorkel_beambell_8376_10403.png snorkel_beambell_10404_15395.png snorkel_beambell_15396_15959.png snorkel_beambell_15960_19763.png snorkel_beambell_19764_21539.png snorkel_beambell_21540_22247.png snorkel_beambell_22248_23291.png snorkel_beambell_23292_23891.png snorkel_beambell_23892_23951.png snorkel_beambell_23952_23999.png snorkel_beambell_24000_24095.png snorkel_beambell_24096_24143.png snorkel_beambell_24144_24191.png snorkel_beambell_24192_26291.png snorkel_beambell_26292_27779.png snorkel_beambell_27780_28151.png snorkel_beambell_28152_29591.png snorkel_beambell_29592_29651.png snorkel_beambell_29652_30599.png snorkel_beambell_30600_30779.png snorkel_beambell_30780_34211.png snorkel_beambell_34212_34331.png snorkel_beambell_34332_34454.png