In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision.
 We propose a rule-exemplar model for collecting human supervision to combine the scalability of rules with the quality of instance labels.   The supervision is coupled such that it is both natural for humans and synergistic for learning. We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables.   Empirical evaluation on five different tasks shows that (1) our algorithm is more accurate than several existing methods of learning from a mix of clean and noisy supervision, and (2)  
the coupled rule-exemplar supervision is effective in denoising rules. With the ever-increasing reach of machine learning, a common hurdle to new adoptions is the lack of labeled data and the pain-staking process involved in collecting it via human supervision. Over the years, several strategies have evolved for reducing the tedium of collecting human supervision. On the one hand are methods like active learning and crowd-consensus learning that seek to reduce the cost of supervision in the form of per-instance labels. On the other hand is the rich history of rule-based methods (Appelt et al., 1993; Cunningham, 2002) where humans code-up their supervision as labeling rules. There is growing interest in learning from such scalable, albiet noisy, supervision (Ratner et al., 2016; Pal & Balasubramanian, 2018; Bach et al., 2019; Sun et al., 2018; Kang et al., 2018 ). However, clean task-specific instance labels continue to be critical for reliable results (Goh et al., 2018; Bach et al., 2019) even when fine-tuning models pre-trained on indirect supervision (Sun et al., 2017; Devlin et al., 2018) . In this paper we propose a unique blend of cheap coarse-grained supervision in the form of rules and expensive fine-grained supervision in the form of labeled instances. Instead of supervising rules and instance labels independently, we propose that each labeling rule be attached with exemplars of where the rule correctly 'fires'. Thus, the rule can be treated as a noisy generalization of those exemplars. Often rules are coded up only after inspecting data. As a human inspects instances, he labels them, and then generalizes them to rules. Thus, humans provide paired supervision of rules and exemplars demonstrating correct deployment of that rule. We explain further with two illustrative applications. Our examples below are from the text domain because rules have been traditionally used in many NLP tasks, but our learning algorithm is agnostic to how rules are expressed. Sentiment Classification Consider an instance I highly recommend this modest priced cellular phone that a human inspects for a sentiment labeling task. After labeling it as positive, he can easily generalize it to a rule Contains 'highly recommend' → positive label. This rule generalizes to several more instances, thereby eliminating the need of per-instance labeling on those. However, the label assigned by this rule on unseen instances may not be as reliable as the explicit label on this specific exemplar it generalized. For example, it misfires on I would highly recommend this phone if it weren't for their poor service. Slot-filling Consider a slot-filling task on restaurant reviews over labels like cuisine, location, and time. When an annotator sees an instance like: what chinese restaurants in this city have good reviews? , after labeling token chinese as cuisine, he generalizes it to a rule: (. * ese|. * ian|mexican) restaurants → (cuisine) restaurants. This rule matches hundreds of instances in the unlabeled set, but could wrongly label a phrase like these restaurants. We present in Section 3 other applications where such supervision is natural. Our focus in this paper is developing algorithms for training models under such coupled rule-exemplar supervision. Our main challenge is that the labels induced by the rules are more noisy than instance-level supervised labels because humans tend to over generalize (Tessler & Goodman, 2019) as we saw in the illustrations above. Learning with noisy labels with or without additional clean data has been a problem of long-standing interest in ML (Khetan et al., 2018; Zhang & Sabuncu, 2018; Ren et al., 2018b; Veit et al., 2017; Shen & Sanghavi, 2019) . However, we seek to design algorithms that better capture rule-specific noise with the help of exemplars around which we have supervision that the rule fired correctly. We associate a latent random variable on whether a rule correctly 'covers' an instance, and jointly learn the distribution among the label and all cover variables. This way we simultaneously train the classifier with corrected rule-label examples, and restrict over-generalized rules. In summary our contributions in this paper are as follows: Our contributions (1) We propose the paradigm of supervision in the form of rules generalizing labeled exemplars that is natural in several applications. (2) We design a training method that simultaneously denoises over-generalized rules via latent coverage variables, and trains a classification model with a soft implication loss that we introduce. (3) Through experiments on five tasks spanning question classification, spam detection, sequence labeling, and record classification we show that our proposed paradigm of supervision enables an effective synergy between rule-level and instance-level supervision. (4) We compare our algorithm to several recent frameworks for learning with noisy supervision and constraints, and show much better results with our method. We proposed a new rule-exemplar model for collecting human supervision to combine the scalability of top-level rules with the quality of instance-level labels. We show that such supervision is natural since humans typically inspect examples to code rules. Furthermore, such coupled examples provide supervision on correct firing of rules which help to denoise rules. We propose to train the classifier while jointly denoising rules via latent coverage variables imposing a soft-implication constraint on the true label. Empirically on five datasets we show that our training algorithm that performs rule-specific denoising is better than generic noise-tolerant learning. In future we plan to deploy this framework on other applications where human supervision is a scarce resource. We model a joint distribution Q(y, r 1 , . . . , r n |x) to capture the interaction among the label random variable y and coverage random variables r 1 , . . . , r n of any instance x. We use r to compactly represent r 1 , . . . , r n . Strictly speaking, when a rule R j does not cover x, the r j is not a random variable and its value is pinned to 0 but we use this fixed-tuple notation for clarity. The random variables r j and y impose a constraint on the joint distribution Q: for a x ∈ H j when r j = 1, the label y cannot be anything other than j . r j = 1 =⇒ y = j ∀x ∈ H j (7) We can convert this into a soft constraint on the marginals of the distribution Q by stating the probability of y = j Q(y, r j = 1|x) should be small. The singleton marginals of Q along the y and r j variables are tied to the P θ and P jφ (r j |x) we seek to learn. A network with parameters θ models the classifier P θ (y|x), and a separate network with φ variables (shared across all rules) learns the P jφ (r j |x) distribution. The marginals of joint Q should match these trained marginals and we use a KL term for that: We call the combined KL term succinctly as KL(Q, P θ ) + KL(Q, P φ ). Further the P θ and P jφ distributions should maximize the log-likelihood on their respective labeled data as provided in Equation 1 and Equation 2 respectively. Putting all the above objectives together with hyper-parameters α > 0, λ > 0 we get our final objective as: We show in Section A.1 that this gives rise to the solution for Q in terms of P θ , P jφ and alternately for P θ , P jφ in terms of Q as follows. where δ(y = j ∧ r j = 1) is an indicator function that is 1 when the constraint inside holds, else it is 0. Computing marginals of the above using straight-forward message passing techniques we get: (13) Thereafter, we solve for θ and φ in terms of a given Q as Here, γ = 1 α . This gives rise to an alternating optimization algorithm as in the posterior regularization framework of Ganchev et al. (2010) . We initialize θ and φ randomly. Then in a loop, we perform the following two steps alternatively much like the EM algorithm (Dempster et al., 1977) .
Coupled rule-exemplar supervision and a implication loss helps to jointly learn to denoise rules and imply labels.
{"src": "scitldr-AIC_train", "paper_id": "SkeuexBtDr"}