Unanswerable
NONE
text classification
EXTRACTIVE
We evaluate the effect batch size has on active learning stopping methods for text classification. We use the publicly available 20Newsgroups dataset in our experiments.
1801.07887
Introduction </s> The Ġuse Ġof Ġactive Ġlearning Ġhas Ġreceived Ġa Ġlot Ġof Ġinterest Ġfor Ġreducing Ġannotation Ġcosts Ġfor Ġtext Ġclassification ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. </s> Active Ġlearning Ġsharply Ġincreases Ġthe Ġperformance Ġof Ġiter atively Ġtrained Ġmachine Ġlearning Ġmodels Ġby Ġselectively Ġdetermining Ġwhich Ġunl abel ed Ġsamples Ġshould Ġbe Ġannot ated . ĠThe Ġnumber Ġof Ġsamples Ġthat Ġare Ġselected Ġfor Ġannotation Ġat Ġeach Ġiteration Ġof Ġactive Ġlearning Ġis Ġcalled Ġthe Ġbatch Ġsize . </s> An Ġimportant Ġaspect Ġof Ġthe Ġactive Ġlearning Ġprocess Ġis Ġwhen Ġto Ġstop Ġthe Ġactive Ġlearning Ġprocess . ĠSto pping Ġmethods Ġenable Ġthe Ġpotential Ġbenefits Ġof Ġactive Ġlearning Ġto Ġbe Ġachieved Ġin Ġpractice . ĠWithout Ġstopping Ġmethods , Ġthe Ġactive Ġlearning Ġprocess Ġwould Ġcontinue Ġuntil Ġall Ġannotations Ġhave Ġbeen Ġlabeled , Ġdefeating Ġthe Ġpurpose Ġof Ġusing Ġactive Ġlearning . ĠAccordingly , Ġthere Ġhas Ġbeen Ġa Ġlot Ġof Ġinterest Ġin Ġthe Ġdevelopment Ġof Ġactive Ġlearning Ġstopping Ġmethods ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> Another Ġimportant Ġaspect Ġof Ġthe Ġactive Ġlearning Ġprocess Ġis Ġwhat Ġbatch Ġsize Ġto Ġuse . ĠPrevious Ġwork Ġhas Ġshown Ġthat Ġusing Ġsmaller Ġbatch Ġsizes Ġleads Ġto Ġgreater Ġlearning Ġefficiency ĠB IB REF 2 Ġ, ĠB IB REF 7 Ġ. ĠThere Ġis Ġa Ġtension Ġbetween Ġusing Ġsmaller Ġbatch Ġsizes Ġto Ġoptimize Ġlearning Ġefficiency Ġand Ġusing Ġlarger Ġbatch Ġsizes Ġto Ġoptimize Ġdevelopment Ġspeed Ġand Ġease Ġof Ġannotation . ĠWe Ġanalyze Ġhow Ġbatch Ġsize Ġaffects Ġa Ġleading Ġstopping Ġmethod Ġand Ġhow Ġstopping Ġmethod Ġparameters Ġcan Ġbe Ġchanged Ġto Ġoptimize Ġperformance Ġdepending Ġon Ġthe Ġbatch Ġsize . </s> We Ġevaluate Ġthe Ġeffect Ġbatch Ġsize Ġhas Ġon Ġactive Ġlearning Ġstopping Ġmethods Ġfor Ġtext Ġclassification . ĠWe Ġuse Ġthe Ġpublicly Ġavailable Ġ20 News groups Ġdataset Ġin Ġour Ġexperiments . </s> For Ġour Ġbase Ġlear ner , Ġwe Ġuse Ġthe Ġimplementation Ġof Ġa ĠSupport ĠVector ĠMachine Ġfrom Ġthe Ġsc ik it - learn ĠPython Ġlibrary . ĠFor Ġour Ġsampling Ġalgorithm , Ġwe Ġuse Ġthe Ġclosest - to - hyper plane Ġalgorithm ĠB IB REF 2 Ġ, Ġwhich Ġhas Ġbeen Ġshown Ġin Ġrecent Ġwork Ġto Ġcompare Ġfavorably Ġwith Ġother Ġsampling Ġalgorithms ĠB IB REF 8 Ġ. ĠWe Ġuse Ġa Ġbinary Ġbag Ġof Ġwords Ġrepresentation Ġand Ġonly Ġconsider Ġwords Ġthat Ġshow Ġup Ġin Ġthe Ġdataset Ġmore Ġthan Ġthree Ġtimes . ĠWe Ġuse Ġa Ġstop Ġword Ġlist Ġto Ġremove Ġcommon ĠEnglish Ġwords . </s> For Ġanalyzing Ġthe Ġimpact Ġof Ġbatch Ġsize Ġon Ġstopping Ġmethods , Ġwe Ġuse Ġa Ġmethod Ġthat Ġwill Ġstop Ġat Ġthe Ġfirst Ġtraining Ġiteration Ġthat Ġis Ġwithin Ġa Ġspecified Ġpercentage Ġof Ġthe Ġmaximum Ġachievable Ġperformance . ĠWe Ġdenote Ġthis Ġmethod Ġas Ġthe ĠOracle ĠMethod , Ġand Ġwe Ġwill Ġset Ġthe Ġpercentage Ġto Ġ99 Ġand Ġdenote Ġthis Ġas ĠOracle - 99 . ĠWe Ġset Ġthe Ġpercentage Ġto Ġ99 Ġbecause Ġit Ġis Ġtypical Ġfor Ġleading Ġstopping Ġmethods Ġto Ġbe Ġable Ġto Ġachieve Ġthis Ġlevel Ġof Ġperformance Ġ( see ĠTable Ġ1 Ġin ĠB IB REF 4 Ġ). ĠAlthough Ġthe ĠOracle ĠMethod Ġcannot Ġbe Ġused Ġin Ġpractice , Ġit Ġis Ġuseful Ġfor Ġcontextual izing Ġthe Ġstopping Ġresults Ġof Ġpractical Ġstopping Ġmethods . </s> Results </s> We Ġconsidered Ġdifferent Ġbatch Ġsizes Ġin Ġour Ġexperiments , Ġbased Ġon Ġpercentages Ġof Ġthe Ġentire Ġset Ġof Ġtraining Ġdata . ĠThe Ġresults Ġfor Ġbatch Ġsizes Ġcorresponding Ġto Ġ1 %, Ġ5 %, Ġand Ġ10 % Ġof Ġthe Ġtraining Ġdata Ġfor Ġthe Ġ20 News groups Ġdataset Ġare Ġsummarized Ġin ĠTable ~ ĠSEC REF 4 Ġ. </s> Oracle ĠResults </s> Looking Ġat ĠTable ~ ĠSEC REF 4 Ġ, Ġone Ġcan Ġsee Ġthat ĠOracle - 99 Ġneeds Ġmore Ġannotations Ġwith Ġlarger Ġbatch Ġper c ents Ġto Ġreach Ġapproximately Ġthe Ġsame ĠF - Measure Ġas Ġwith Ġsmaller Ġbatch Ġper c ents . </s> These Ġresults Ġare Ġconsistent Ġwith Ġpast Ġfindings Ġthat Ġlearning Ġefficiency Ġis Ġdecreased Ġwith Ġlarger Ġbatch Ġsizes ĠB IB REF 2 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġan Ġopen Ġquestion Ġis Ġwhether Ġchanging Ġthe Ġparameters Ġassociated Ġwith Ġactual Ġstopping Ġmethods Ġcan Ġmake Ġthem Ġexperience Ġless Ġdegradation Ġin Ġperformance Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . ĠIn Ġparticular , Ġan Ġimportant Ġparameter Ġof Ġstopping Ġmethods Ġis Ġthe Ġwindow Ġsize Ġof Ġprevious Ġiterations Ġto Ġconsider . ĠThe Ġnext Ġsubsection Ġshows Ġhow Ġdecreasing Ġthe Ġwindow Ġsize Ġparameter Ġcan Ġhelp Ġto Ġreduce Ġthe Ġdegradation Ġin Ġperformance Ġthat Ġstopping Ġmethods Ġexperience Ġwith Ġlarger Ġbatch Ġsizes . </s> Comp aring ĠB V 2009 Ġwith Ġthe ĠOracle ĠMethod </s> We Ġdenote Ġthe Ġstopping Ġmethod Ġpublished Ġin ĠB IB REF 4 Ġas ĠB V 2009 . ĠThis Ġstopping Ġmethod Ġwill Ġstop Ġthe Ġactive Ġlearning Ġprocess Ġif Ġthe Ġmean Ġof Ġthe Ġthree Ġprevious Ġk appa Ġagreement Ġvalues Ġbetween Ġconsecutive Ġmodels Ġis Ġabove Ġa Ġthreshold . ĠFor Ġlarger Ġbatch Ġper c ents , Ġnote Ġthat ĠB V 2009 Ġstops Ġlater Ġthan Ġthe Ġoptimal ĠOracle ĠMethod Ġpoint . </s> We Ġran ĠB V 2009 Ġwith Ġsmaller Ġwindow Ġsizes Ġfor Ġeach Ġof Ġour Ġdifferent Ġbatch Ġsizes . ĠOur Ġresults Ġare Ġsummarized Ġfor Ġa Ġwindow Ġsize Ġof Ġone Ġin Ġthe Ġrow Ġ`` B V 2009 Ġ( Window ĠSize Ġ= Ġ1 ) '' Ġin ĠTable ~ ĠSEC REF 4 Ġ. ĠWhen Ġusing Ġa Ġwindow Ġsize Ġof Ġone , ĠB V 2009 Ġis Ġable Ġto Ġstop Ġwith Ġa Ġsmaller Ġnumber Ġof Ġannotations Ġthan Ġwhen Ġusing Ġa Ġwindow Ġsize Ġof Ġthree . ĠThis Ġis Ġdone Ġwithout Ġlosing Ġmuch ĠF - Measure . ĠThe Ġnext Ġsubsection Ġprovides Ġan Ġexplanation Ġas Ġto Ġwhy Ġsmaller Ġwindow Ġsizes Ġare Ġmore Ġeffective Ġthan Ġlarger Ġwindow Ġsizes Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . </s> B V 2009 ĠWindow ĠSize ĠDiscussion </s> We Ġset ĠIN LINE FORM 0 Ġto Ġbe Ġthe Ġwindow Ġsize Ġthat Ġthe Ġuser Ġhas Ġdefined . ĠKappa Ġis Ġan Ġagreement Ġmetric Ġbetween Ġtwo Ġmodels . ĠTherefore , ĠB V 2009 Ġneeds ĠIN LINE FORM 1 Ġmodels Ġto Ġbe Ġgenerated Ġbefore Ġit Ġbegins Ġto Ġcheck Ġif Ġthe Ġaverage Ġis Ġabove Ġthe Ġthreshold . ĠThis Ġdoes Ġnot Ġnecessarily Ġmean Ġit Ġstops Ġafter ĠIN LINE FORM 2 Ġmodels Ġhave Ġbeen Ġgenerated . ĠRather , Ġit Ġrepresents Ġthe Ġfirst Ġpoint Ġin Ġthe Ġactive Ġlearning Ġprocess Ġat Ġwhich ĠB V 2009 Ġeven Ġhas Ġa Ġchance Ġto Ġstop . </s> When Ġusing Ġlarger Ġbatch Ġper c ents , Ġfewer Ġmodels Ġare Ġgenerated Ġthan Ġwhen Ġusing Ġsmaller Ġbatch Ġper c ents . ĠThis Ġgives Ġany Ġstopping Ġmethod Ġless Ġpoints Ġto Ġtest Ġwhether Ġor Ġnot Ġto Ġstop . ĠWe Ġalso Ġnote Ġthat Ġk appa Ġagreement Ġscores Ġare Ġgenerally Ġlow Ġbetween Ġthe Ġfirst Ġfew Ġmodels Ġtrained . ĠThis , Ġcombined Ġwith Ġfewer Ġpoints Ġto Ġstop Ġat , Ġcauses ĠB V 2009 Ġto Ġstop Ġsomewhat Ġsub - optim ally Ġwhen Ġusing Ġvery Ġlarge Ġbatch Ġper c ents . ĠUsage Ġof Ġvery Ġlarge Ġbatch Ġsizes , Ġsuch Ġas Ġ10 % Ġof Ġthe Ġdata , Ġis Ġnot Ġcommon Ġso Ġsub - opt imal Ġperformance Ġof Ġstopping Ġmethods Ġin Ġthose Ġsituations Ġis Ġnot Ġa Ġmajor Ġproblem . </s> Conclusion </s> Active Ġlearning Ġhas Ġthe Ġpotential Ġto Ġsignificantly Ġreduce Ġannotation Ġcosts . ĠTwo Ġimportant Ġconsiderations Ġin Ġthe Ġactive Ġlearning Ġprocess Ġare Ġwhen Ġto Ġstop Ġthe Ġiter ative Ġprocess Ġof Ġasking Ġfor Ġmore Ġlabeled Ġdata Ġand Ġhow Ġlarge Ġof Ġa Ġbatch Ġsize Ġto Ġuse Ġwhen Ġasking Ġfor Ġadditional Ġlabels Ġduring Ġeach Ġiteration . ĠWe Ġfound Ġthat Ġstopping Ġmethods Ġdegrade Ġin Ġperformance Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . ĠThe Ġdegradation Ġin Ġperformance Ġis Ġlarger Ġthan Ġthe Ġamount Ġthat Ġcan Ġbe Ġexplained Ġdue Ġto Ġthe Ġdegradation Ġin Ġlearning Ġefficiency Ġthat Ġresults Ġfrom Ġusing Ġlarger Ġbatch Ġsizes . ĠAn Ġimportant Ġparameter Ġused Ġby Ġstopping Ġmethods Ġis Ġwhat Ġwindow Ġsize Ġof Ġearlier Ġiterations Ġto Ġconsider Ġin Ġmaking Ġthe Ġstopping Ġdecision . ĠOur Ġresults Ġindicate Ġthat Ġmaking Ġthe Ġwindow Ġsize Ġsmaller Ġhelps Ġto Ġmitigate Ġthe Ġdegradation Ġin Ġstopping Ġmethod Ġperformance Ġthat Ġoccurs Ġwith Ġlarger Ġbatch Ġsizes . </s> Acknowled gment </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠThe ĠCollege Ġof ĠNew ĠJersey ĠSupport Ġof ĠScholar ly ĠActivities Ġ( S OS A ) Ġprogram , Ġby ĠThe ĠCollege Ġof ĠNew ĠJersey ĠMent ored ĠUnder graduate ĠSummer ĠExperience Ġ( M USE ) Ġprogram , Ġand Ġby Ġusage Ġof ĠThe ĠCollege Ġof ĠNew ĠJersey ĠHigh ĠPerformance ĠComputing ĠSystem .
Introduction The use of active learning has received a lot of interest for reducing annotation costs for text classification BIBREF0 , BIBREF1 , BIBREF2 . Active learning sharply increases the performance of iteratively trained machine learning models by selectively determining which unlabeled samples should be annotated. The number of samples that are selected for annotation at each iteration of active learning is called the batch size. An important aspect of the active learning process is when to stop the active learning process. Stopping methods enable the potential benefits of active learning to be achieved in practice. Without stopping methods, the active learning process would continue until all annotations have been labeled, defeating the purpose of using active learning. Accordingly, there has been a lot of interest in the development of active learning stopping methods BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . Another important aspect of the active learning process is what batch size to use. Previous work has shown that using smaller batch sizes leads to greater learning efficiency BIBREF2 , BIBREF7 . There is a tension between using smaller batch sizes to optimize learning efficiency and using larger batch sizes to optimize development speed and ease of annotation. We analyze how batch size affects a leading stopping method and how stopping method parameters can be changed to optimize performance depending on the batch size. We evaluate the effect batch size has on active learning stopping methods for text classification. We use the publicly available 20Newsgroups dataset in our experiments. For our base learner, we use the implementation of a Support Vector Machine from the scikit-learn Python library. For our sampling algorithm, we use the closest-to-hyperplane algorithm BIBREF2 , which has been shown in recent work to compare favorably with other sampling algorithms BIBREF8 . We use a binary bag of words representation and only consider words that show up in the dataset more than three times. We use a stop word list to remove common English words. For analyzing the impact of batch size on stopping methods, we use a method that will stop at the first training iteration that is within a specified percentage of the maximum achievable performance. We denote this method as the Oracle Method, and we will set the percentage to 99 and denote this as Oracle-99. We set the percentage to 99 because it is typical for leading stopping methods to be able to achieve this level of performance (see Table 1 in BIBREF4 ). Although the Oracle Method cannot be used in practice, it is useful for contextualizing the stopping results of practical stopping methods. Results We considered different batch sizes in our experiments, based on percentages of the entire set of training data. The results for batch sizes corresponding to 1%, 5%, and 10% of the training data for the 20Newsgroups dataset are summarized in Table~ SECREF4 . Oracle Results Looking at Table~ SECREF4 , one can see that Oracle-99 needs more annotations with larger batch percents to reach approximately the same F-Measure as with smaller batch percents. These results are consistent with past findings that learning efficiency is decreased with larger batch sizes BIBREF2 , BIBREF7 . However, an open question is whether changing the parameters associated with actual stopping methods can make them experience less degradation in performance when larger batch sizes are used. In particular, an important parameter of stopping methods is the window size of previous iterations to consider. The next subsection shows how decreasing the window size parameter can help to reduce the degradation in performance that stopping methods experience with larger batch sizes. Comparing BV2009 with the Oracle Method We denote the stopping method published in BIBREF4 as BV2009. This stopping method will stop the active learning process if the mean of the three previous kappa agreement values between consecutive models is above a threshold. For larger batch percents, note that BV2009 stops later than the optimal Oracle Method point. We ran BV2009 with smaller window sizes for each of our different batch sizes. Our results are summarized for a window size of one in the row ``BV2009 (Window Size = 1)'' in Table~ SECREF4 . When using a window size of one, BV2009 is able to stop with a smaller number of annotations than when using a window size of three. This is done without losing much F-Measure. The next subsection provides an explanation as to why smaller window sizes are more effective than larger window sizes when larger batch sizes are used. BV2009 Window Size Discussion We set INLINEFORM0 to be the window size that the user has defined. Kappa is an agreement metric between two models. Therefore, BV2009 needs INLINEFORM1 models to be generated before it begins to check if the average is above the threshold. This does not necessarily mean it stops after INLINEFORM2 models have been generated. Rather, it represents the first point in the active learning process at which BV2009 even has a chance to stop. When using larger batch percents, fewer models are generated than when using smaller batch percents. This gives any stopping method less points to test whether or not to stop. We also note that kappa agreement scores are generally low between the first few models trained. This, combined with fewer points to stop at, causes BV2009 to stop somewhat sub-optimally when using very large batch percents. Usage of very large batch sizes, such as 10% of the data, is not common so sub-optimal performance of stopping methods in those situations is not a major problem. Conclusion Active learning has the potential to significantly reduce annotation costs. Two important considerations in the active learning process are when to stop the iterative process of asking for more labeled data and how large of a batch size to use when asking for additional labels during each iteration. We found that stopping methods degrade in performance when larger batch sizes are used. The degradation in performance is larger than the amount that can be explained due to the degradation in learning efficiency that results from using larger batch sizes. An important parameter used by stopping methods is what window size of earlier iterations to consider in making the stopping decision. Our results indicate that making the window size smaller helps to mitigate the degradation in stopping method performance that occurs with larger batch sizes. Acknowledgment This work was supported in part by The College of New Jersey Support of Scholarly Activities (SOSA) program, by The College of New Jersey Mentored Undergraduate Summer Experience (MUSE) program, and by usage of The College of New Jersey High Performance Computing System.
What downstream tasks are evaluated?
f67b9bda14ec70feba2e0d10c400b2b2025a0a6a
What Ġdownstream Ġtasks Ġare Ġevaluated ?
Un answer able
<s> What Ġdownstream Ġtasks Ġare Ġevaluated ? </s> Introduction </s> The Ġuse Ġof Ġactive Ġlearning Ġhas Ġreceived Ġa Ġlot Ġof Ġinterest Ġfor Ġreducing Ġannotation Ġcosts Ġfor Ġtext Ġclassification ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. </s> Active Ġlearning Ġsharply Ġincreases Ġthe Ġperformance Ġof Ġiter atively Ġtrained Ġmachine Ġlearning Ġmodels Ġby Ġselectively Ġdetermining Ġwhich Ġunl abel ed Ġsamples Ġshould Ġbe Ġannot ated . ĠThe Ġnumber Ġof Ġsamples Ġthat Ġare Ġselected Ġfor Ġannotation Ġat Ġeach Ġiteration Ġof Ġactive Ġlearning Ġis Ġcalled Ġthe Ġbatch Ġsize . </s> An Ġimportant Ġaspect Ġof Ġthe Ġactive Ġlearning Ġprocess Ġis Ġwhen Ġto Ġstop Ġthe Ġactive Ġlearning Ġprocess . ĠSto pping Ġmethods Ġenable Ġthe Ġpotential Ġbenefits Ġof Ġactive Ġlearning Ġto Ġbe Ġachieved Ġin Ġpractice . ĠWithout Ġstopping Ġmethods , Ġthe Ġactive Ġlearning Ġprocess Ġwould Ġcontinue Ġuntil Ġall Ġannotations Ġhave Ġbeen Ġlabeled , Ġdefeating Ġthe Ġpurpose Ġof Ġusing Ġactive Ġlearning . ĠAccordingly , Ġthere Ġhas Ġbeen Ġa Ġlot Ġof Ġinterest Ġin Ġthe Ġdevelopment Ġof Ġactive Ġlearning Ġstopping Ġmethods ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> Another Ġimportant Ġaspect Ġof Ġthe Ġactive Ġlearning Ġprocess Ġis Ġwhat Ġbatch Ġsize Ġto Ġuse . ĠPrevious Ġwork Ġhas Ġshown Ġthat Ġusing Ġsmaller Ġbatch Ġsizes Ġleads Ġto Ġgreater Ġlearning Ġefficiency ĠB IB REF 2 Ġ, ĠB IB REF 7 Ġ. ĠThere Ġis Ġa Ġtension Ġbetween Ġusing Ġsmaller Ġbatch Ġsizes Ġto Ġoptimize Ġlearning Ġefficiency Ġand Ġusing Ġlarger Ġbatch Ġsizes Ġto Ġoptimize Ġdevelopment Ġspeed Ġand Ġease Ġof Ġannotation . ĠWe Ġanalyze Ġhow Ġbatch Ġsize Ġaffects Ġa Ġleading Ġstopping Ġmethod Ġand Ġhow Ġstopping Ġmethod Ġparameters Ġcan Ġbe Ġchanged Ġto Ġoptimize Ġperformance Ġdepending Ġon Ġthe Ġbatch Ġsize . </s> We Ġevaluate Ġthe Ġeffect Ġbatch Ġsize Ġhas Ġon Ġactive Ġlearning Ġstopping Ġmethods Ġfor Ġtext Ġclassification . ĠWe Ġuse Ġthe Ġpublicly Ġavailable Ġ20 News groups Ġdataset Ġin Ġour Ġexperiments . </s> For Ġour Ġbase Ġlear ner , Ġwe Ġuse Ġthe Ġimplementation Ġof Ġa ĠSupport ĠVector ĠMachine Ġfrom Ġthe Ġsc ik it - learn ĠPython Ġlibrary . ĠFor Ġour Ġsampling Ġalgorithm , Ġwe Ġuse Ġthe Ġclosest - to - hyper plane Ġalgorithm ĠB IB REF 2 Ġ, Ġwhich Ġhas Ġbeen Ġshown Ġin Ġrecent Ġwork Ġto Ġcompare Ġfavorably Ġwith Ġother Ġsampling Ġalgorithms ĠB IB REF 8 Ġ. ĠWe Ġuse Ġa Ġbinary Ġbag Ġof Ġwords Ġrepresentation Ġand Ġonly Ġconsider Ġwords Ġthat Ġshow Ġup Ġin Ġthe Ġdataset Ġmore Ġthan Ġthree Ġtimes . ĠWe Ġuse Ġa Ġstop Ġword Ġlist Ġto Ġremove Ġcommon ĠEnglish Ġwords . </s> For Ġanalyzing Ġthe Ġimpact Ġof Ġbatch Ġsize Ġon Ġstopping Ġmethods , Ġwe Ġuse Ġa Ġmethod Ġthat Ġwill Ġstop Ġat Ġthe Ġfirst Ġtraining Ġiteration Ġthat Ġis Ġwithin Ġa Ġspecified Ġpercentage Ġof Ġthe Ġmaximum Ġachievable Ġperformance . ĠWe Ġdenote Ġthis Ġmethod Ġas Ġthe ĠOracle ĠMethod , Ġand Ġwe Ġwill Ġset Ġthe Ġpercentage Ġto Ġ99 Ġand Ġdenote Ġthis Ġas ĠOracle - 99 . ĠWe Ġset Ġthe Ġpercentage Ġto Ġ99 Ġbecause Ġit Ġis Ġtypical Ġfor Ġleading Ġstopping Ġmethods Ġto Ġbe Ġable Ġto Ġachieve Ġthis Ġlevel Ġof Ġperformance Ġ( see ĠTable Ġ1 Ġin ĠB IB REF 4 Ġ). ĠAlthough Ġthe ĠOracle ĠMethod Ġcannot Ġbe Ġused Ġin Ġpractice , Ġit Ġis Ġuseful Ġfor Ġcontextual izing Ġthe Ġstopping Ġresults Ġof Ġpractical Ġstopping Ġmethods . </s> Results </s> We Ġconsidered Ġdifferent Ġbatch Ġsizes Ġin Ġour Ġexperiments , Ġbased Ġon Ġpercentages Ġof Ġthe Ġentire Ġset Ġof Ġtraining Ġdata . ĠThe Ġresults Ġfor Ġbatch Ġsizes Ġcorresponding Ġto Ġ1 %, Ġ5 %, Ġand Ġ10 % Ġof Ġthe Ġtraining Ġdata Ġfor Ġthe Ġ20 News groups Ġdataset Ġare Ġsummarized Ġin ĠTable ~ ĠSEC REF 4 Ġ. </s> Oracle ĠResults </s> Looking Ġat ĠTable ~ ĠSEC REF 4 Ġ, Ġone Ġcan Ġsee Ġthat ĠOracle - 99 Ġneeds Ġmore Ġannotations Ġwith Ġlarger Ġbatch Ġper c ents Ġto Ġreach Ġapproximately Ġthe Ġsame ĠF - Measure Ġas Ġwith Ġsmaller Ġbatch Ġper c ents . </s> These Ġresults Ġare Ġconsistent Ġwith Ġpast Ġfindings Ġthat Ġlearning Ġefficiency Ġis Ġdecreased Ġwith Ġlarger Ġbatch Ġsizes ĠB IB REF 2 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġan Ġopen Ġquestion Ġis Ġwhether Ġchanging Ġthe Ġparameters Ġassociated Ġwith Ġactual Ġstopping Ġmethods Ġcan Ġmake Ġthem Ġexperience Ġless Ġdegradation Ġin Ġperformance Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . ĠIn Ġparticular , Ġan Ġimportant Ġparameter Ġof Ġstopping Ġmethods Ġis Ġthe Ġwindow Ġsize Ġof Ġprevious Ġiterations Ġto Ġconsider . ĠThe Ġnext Ġsubsection Ġshows Ġhow Ġdecreasing Ġthe Ġwindow Ġsize Ġparameter Ġcan Ġhelp Ġto Ġreduce Ġthe Ġdegradation Ġin Ġperformance Ġthat Ġstopping Ġmethods Ġexperience Ġwith Ġlarger Ġbatch Ġsizes . </s> Comp aring ĠB V 2009 Ġwith Ġthe ĠOracle ĠMethod </s> We Ġdenote Ġthe Ġstopping Ġmethod Ġpublished Ġin ĠB IB REF 4 Ġas ĠB V 2009 . ĠThis Ġstopping Ġmethod Ġwill Ġstop Ġthe Ġactive Ġlearning Ġprocess Ġif Ġthe Ġmean Ġof Ġthe Ġthree Ġprevious Ġk appa Ġagreement Ġvalues Ġbetween Ġconsecutive Ġmodels Ġis Ġabove Ġa Ġthreshold . ĠFor Ġlarger Ġbatch Ġper c ents , Ġnote Ġthat ĠB V 2009 Ġstops Ġlater Ġthan Ġthe Ġoptimal ĠOracle ĠMethod Ġpoint . </s> We Ġran ĠB V 2009 Ġwith Ġsmaller Ġwindow Ġsizes Ġfor Ġeach Ġof Ġour Ġdifferent Ġbatch Ġsizes . ĠOur Ġresults Ġare Ġsummarized Ġfor Ġa Ġwindow Ġsize Ġof Ġone Ġin Ġthe Ġrow Ġ`` B V 2009 Ġ( Window ĠSize Ġ= Ġ1 ) '' Ġin ĠTable ~ ĠSEC REF 4 Ġ. ĠWhen Ġusing Ġa Ġwindow Ġsize Ġof Ġone , ĠB V 2009 Ġis Ġable Ġto Ġstop Ġwith Ġa Ġsmaller Ġnumber Ġof Ġannotations Ġthan Ġwhen Ġusing Ġa Ġwindow Ġsize Ġof Ġthree . ĠThis Ġis Ġdone Ġwithout Ġlosing Ġmuch ĠF - Measure . ĠThe Ġnext Ġsubsection Ġprovides Ġan Ġexplanation Ġas Ġto Ġwhy Ġsmaller Ġwindow Ġsizes Ġare Ġmore Ġeffective Ġthan Ġlarger Ġwindow Ġsizes Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . </s> B V 2009 ĠWindow ĠSize ĠDiscussion </s> We Ġset ĠIN LINE FORM 0 Ġto Ġbe Ġthe Ġwindow Ġsize Ġthat Ġthe Ġuser Ġhas Ġdefined . ĠKappa Ġis Ġan Ġagreement Ġmetric Ġbetween Ġtwo Ġmodels . ĠTherefore , ĠB V 2009 Ġneeds ĠIN LINE FORM 1 Ġmodels Ġto Ġbe Ġgenerated Ġbefore Ġit Ġbegins Ġto Ġcheck Ġif Ġthe Ġaverage Ġis Ġabove Ġthe Ġthreshold . ĠThis Ġdoes Ġnot Ġnecessarily Ġmean Ġit Ġstops Ġafter ĠIN LINE FORM 2 Ġmodels Ġhave Ġbeen Ġgenerated . ĠRather , Ġit Ġrepresents Ġthe Ġfirst Ġpoint Ġin Ġthe Ġactive Ġlearning Ġprocess Ġat Ġwhich ĠB V 2009 Ġeven Ġhas Ġa Ġchance Ġto Ġstop . </s> When Ġusing Ġlarger Ġbatch Ġper c ents , Ġfewer Ġmodels Ġare Ġgenerated Ġthan Ġwhen Ġusing Ġsmaller Ġbatch Ġper c ents . ĠThis Ġgives Ġany Ġstopping Ġmethod Ġless Ġpoints Ġto Ġtest Ġwhether Ġor Ġnot Ġto Ġstop . ĠWe Ġalso Ġnote Ġthat Ġk appa Ġagreement Ġscores Ġare Ġgenerally Ġlow Ġbetween Ġthe Ġfirst Ġfew Ġmodels Ġtrained . ĠThis , Ġcombined Ġwith Ġfewer Ġpoints Ġto Ġstop Ġat , Ġcauses ĠB V 2009 Ġto Ġstop Ġsomewhat Ġsub - optim ally Ġwhen Ġusing Ġvery Ġlarge Ġbatch Ġper c ents . ĠUsage Ġof Ġvery Ġlarge Ġbatch Ġsizes , Ġsuch Ġas Ġ10 % Ġof Ġthe Ġdata , Ġis Ġnot Ġcommon Ġso Ġsub - opt imal Ġperformance Ġof Ġstopping Ġmethods Ġin Ġthose Ġsituations Ġis Ġnot Ġa Ġmajor Ġproblem . </s> Conclusion </s> Active Ġlearning Ġhas Ġthe Ġpotential Ġto Ġsignificantly Ġreduce Ġannotation Ġcosts . ĠTwo Ġimportant Ġconsiderations Ġin Ġthe Ġactive Ġlearning Ġprocess Ġare Ġwhen Ġto Ġstop Ġthe Ġiter ative Ġprocess Ġof Ġasking Ġfor Ġmore Ġlabeled Ġdata Ġand Ġhow Ġlarge Ġof Ġa Ġbatch Ġsize Ġto Ġuse Ġwhen Ġasking Ġfor Ġadditional Ġlabels Ġduring Ġeach Ġiteration . ĠWe Ġfound Ġthat Ġstopping Ġmethods Ġdegrade Ġin Ġperformance Ġwhen Ġlarger Ġbatch Ġsizes Ġare Ġused . ĠThe Ġdegradation Ġin Ġperformance Ġis Ġlarger Ġthan Ġthe Ġamount Ġthat Ġcan Ġbe Ġexplained Ġdue Ġto Ġthe Ġdegradation Ġin Ġlearning Ġefficiency Ġthat Ġresults Ġfrom Ġusing Ġlarger Ġbatch Ġsizes . ĠAn Ġimportant Ġparameter Ġused Ġby Ġstopping Ġmethods Ġis Ġwhat Ġwindow Ġsize Ġof Ġearlier Ġiterations Ġto Ġconsider Ġin Ġmaking Ġthe Ġstopping Ġdecision . ĠOur Ġresults Ġindicate Ġthat Ġmaking Ġthe Ġwindow Ġsize Ġsmaller Ġhelps Ġto Ġmitigate Ġthe Ġdegradation Ġin Ġstopping Ġmethod Ġperformance Ġthat Ġoccurs Ġwith Ġlarger Ġbatch Ġsizes . </s> Acknowled gment </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠThe ĠCollege Ġof ĠNew ĠJersey ĠSupport Ġof ĠScholar ly ĠActivities Ġ( S OS A ) Ġprogram , Ġby ĠThe ĠCollege Ġof ĠNew ĠJersey ĠMent ored ĠUnder graduate ĠSummer ĠExperience Ġ( M USE ) Ġprogram , Ġand Ġby Ġusage Ġof ĠThe ĠCollege Ġof ĠNew ĠJersey ĠHigh ĠPerformance ĠComputing ĠSystem .