<commit_before><div class="guidedInner">
  <uib-progressbar ng-if="currentStep < 7" max="6" value="currentStep">{{ctrl.getPercent(6) | number:0}}%</uib-progressbar>
  <uib-progressbar ng-if="currentStep > 6" max="9" value="currentStep">{{ctrl.getPercent(9) | number:0}}%</uib-progressbar>
  <div id="guided4" style="margin-top: 30px">
    <span ng-show="currentStep == 0" class="guidedQuestion">Do you want to remove <span uib-tooltip="Words below a certain occurrence"><strong>rare words</strong></span>?</span>
    <span ng-show="currentStep == 1" class="guidedQuestion">Do you want to import a list of <span uib-tooltip="Each word in the list will be removed from the dataset"><strong>words to remove</strong></span>?</span>
    <span ng-show="currentStep == 2" class="guidedQuestion">Do you want to import a list of <span uib-tooltip="Import your Dictionary with correct words here"><strong>correct words</strong></span>?</span>
    <span ng-show="currentStep == 3" class="guidedQuestion">Do you want to reduce <span uib-tooltip="Like - favourite, favorite, favourites or favorites"><strong>word forms</strong></span> and <span uib-tooltip="Misspelled words like - necesary"><strong>spelling errors</strong></span>?</span>
    <span ng-show="currentStep == 4" class="guidedQuestion">Do you want to find <span uib-tooltip="A tag might be composed of multiple words like - hard rock"><strong>multiword tags</strong></span> which appear very <span uib-tooltip="Frequent multiword tags are as example genres"><strong>often</strong></span>?</span>
    <span ng-show="currentStep == 5" class="guidedQuestion">Do you want to find <span uib-tooltip="A tag might be composed of multiple words like - hard rock"><strong>multiword tags</strong></span> which are <span uib-tooltip="Unique multiword tags are as example artist or band names"><strong>unique</strong></span>?</span>
    <span ng-show="currentStep == 6" class="guidedQuestion">Results</span>
    <span ng-show="currentStep == 7" class="guidedQuestion">Do you want to reduce the total amount of <span uib-tooltip="Filter the tags by there tag quality to remove bad tags"><strong>important words</strong></span> used in your final dataset?</span>
    <span ng-show="currentStep == 8" class="guidedQuestion">Do you want to change those <span uib-tooltip="Remove bad words or translate words or resolve synonyms"><strong>import tags</strong></span> by hand?</span>
    <span ng-show="currentStep == 9" class="guidedQuestion">Results</span>
  </div>
  <br>
  <button ng-if="currentStep > 0" class="btn btn-back" ng-click="ctrl.back()"><i class="fa fa-chevron-left" style="position: relative; top:-2px"></i></button>
  <div class="btn-group" ng-if="currentStep != 6 && currentStep != 9">
    <button id="guided3" ng-click="ctrl.customize()" class="btn secondaryColor" style="width: 150px">Customize <span class="fa fa-gears"></span></button>

    <button id="guided1" ng-if="currentStep < 6" ng-click="ctrl.default()" class="btn primaryColor" style="width: 150px">Use Default <span class="glyphicon glyphicon-play"></span></button>
  </div>
  <br>
  <button ng-if="showStep" ng-click="ctrl.ok()" class="btn primaryColor" style="width: 150px; margin-top: 2px">Apply <span class="glyphicon glyphicon-ok"></span></button>
  <div class="btn-group" ng-if="currentStep == 6 || currentStep == 9">
    <button ng-if="currentStep == 9" ng-click="ctrl.restartFinalize()" class="btn secondaryColor" style="width: 100px">Refine <span class="fa fa-flask"></span></button>
    <button id="result4" ng-if="currentStep < 9" ng-click="ctrl.goToLinked()" class="btn secondaryColor" style="width: 100px">Refine <span class="fa fa-bar-chart"></span></button>
    <button id="result5" ng-click="ctrl.goToAdvanced()" class="btn primaryColor">Advanced Mode <span class="fa fa-list-alt"></span></button>
    <button id="result6" ng-if="currentStep < 9" ng-click="ctrl.ok()" class="btn secondaryColor" style="width: 100px">Finalize <span class="fa fa-flask"></span></button>
  </div>
  <hr>
  <div style="position: relative; height: 100%">
  <div ng-if="currentStep == 0 && ctrl.initRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 3 && ctrl.preRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 4 && ctrl.spellRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 5 && ctrl.spellRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 6 && ctrl.compRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
    <div ng-if="currentStep == 9 && ctrl.postRunning" class="overlayGuided">
      <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
    </div>
  <div uib-collapse="!showStep" style="text-align: left" class="guidedInner">
    <div ng-show="currentStep == 0">
      <div class="row" ng-controller="PreprocessingfilterCtrl as ctrl" ng-include="'views/preprocessing.filter.html'"></div>
    </div>
    <div ng-show="currentStep == 1">
      <div class="row" ng-controller="PreprocessingimportCtrl as ctrl" ng-include="'views/preprocessing.import.html'"></div>
    </div>
    <div ng-show="currentStep == 2">
      <div class="row" ng-controller="SpellcheckingimportCtrl as ctrl" ng-include="'views/spellchecking.import.html'"></div>
    </div>
    <div ng-show="currentStep == 3">
      <div class="row" ng-controller="SpellcheckingsetCtrl as ctrl" ng-include="'views/spellchecking.set.html'"></div>
    </div>
    <div ng-show="currentStep == 4">
      <div class="row" ng-controller="CompositefrequentCtrl as ctrl" ng-include="'views/composite.frequent.html'"></div>
    </div>
    <div ng-show="currentStep == 5">
      <div class="row" ng-controller="CompositeuniqueCtrl	 as ctrl" ng-include="'views/composite.unique.html'"></div>
    </div>
    <div ng-show="currentStep == 6">
      <div class="row" ng-controller="OverviewCtrl as ctrl" ng-include="'views/overview.html'"></div>
    </div>
    <div ng-show="currentStep == 7">
      <div class="row" ng-controller="PostprocessingimportantCtrl as ctrl" ng-include="'views/postprocessing.important.html'"></div>
    </div>
    <div ng-show="currentStep == 8">
      <div class="row" ng-controller="PostprocessingreplaceCtrl as ctrl" ng-include="'views/postprocessing.replace.html'"></div>
    </div>
    <div ng-show="currentStep == 9">
      <div class="row" ng-controller="OverviewCtrl as ctrl" ng-include="'views/overview.html'"></div>
    </div>
  </div>
    <div id="guided2" ng-if="!showStep">
      <div ng-if="currentStep == 0" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Rare Words - Default</h2>
          <p>By default <strong>no words </strong>will be removed. The removal of words is necessary if your dataset is big and uses a lot of computation time.</p>
          <p>You might wanna start with a small dataset to find good parameters - and add the filtered ones later.</p>
        </div>
      </div>
      <div ng-if="currentStep == 1" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Blacklist - Default</h2>
          <p>By default <span uib-tooltip="Like: and, or, the, ..."><strong>English stopwords </strong></span> and <span class="descMarker" uib-tooltip="Like: a, b, ... , 1, 2, ..."> single characters</span> will be removed. This reduces computation time and might improve the detection of meaningful single word tags.</p>
          <p>On the other hand stopwords can be an integral part of multiword tags, e.g. "Rock and Roll".</p>
        </div>
      </div>
      <div ng-if="currentStep == 2" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Dictionary - Default</h2>
          <p>By default the <span class="descMarker" uib-tooltip="The most important words for second language learners of English.">General Service List </span>will be imported.</p>
          <p>In small datasets in particular, you might want to import your own list, e.g. genres, to <span uib-tooltip="First the algorithm looks for similar words to the importet dictionary."><strong>improve the quality</strong></span> of the spell correction algorithm.</p>
        </div>
      </div>
      <div ng-if="currentStep == 3" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Spell Correction - Default</h2>
          <p>By default, words with a <span uib-tooltip="Is computed from the intial weights"><strong>word quality</strong></span> above <strong>0.7 </strong> will be treated as correct and words with a similarity above <strong> 0.7</strong> will be replaced by the word with the higher quality.</p>
        </div>
      </div>
      <div ng-if="currentStep == 4" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Frequent Multiword Tags - Default</h2>
          <p>By default a group strength of <strong>0.35 </strong>is used.</p>
          <p>1 Means this group appeared most often and 0 it appeared never.</p>
        </div>
      </div>
      <div ng-if="currentStep == 5" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Unique Multiword Tags - Default</h2>
          <p>By default a group strength of <strong>0.7 </strong>will be used.</p>
          <p>1 Means this two words occur very often together an are less separated and 0 means the words are never together and often separated</p>
        </div>
      </div>
      <div ng-if="currentStep == 7" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Final Filtering</h2>
          <p>Select a set of <strong>important tags</strong>.</p>
          <p>Only the select tags <span class="descMarker" uib-tooltip="There is a option in the advanced mode to use all tags at the end but only the selected ones for the following salvage step.">remain</span> in the dataset.</p>
          <p>The following step searches in the tags below the selected threshold for the words above or equal the threshold.</p>
        </div>
      </div>
      <div ng-if="currentStep == 8" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Salvager</h2>
          <p>You can change or remove tags from the selected tags. You might want to remove non-informative words or fix synonyms by hand.</p>
          <p>Afterwards the salvager searches for the updated important tags in the not important tags. This increases the total dataset coverage.</p>
        </div>
      </div>
    </div>
  </div>
</div>
<commit_msg>Updated texts in step 0<commit_after><div class="guidedInner">
  <uib-progressbar ng-if="currentStep < 7" max="6" value="currentStep">{{ctrl.getPercent(6) | number:0}}%</uib-progressbar>
  <uib-progressbar ng-if="currentStep > 6" max="9" value="currentStep">{{ctrl.getPercent(9) | number:0}}%</uib-progressbar>
  <div id="guided4" style="margin-top: 30px">
    <span ng-show="currentStep == 0" class="guidedQuestion">Do you want to remove <span uib-tooltip="As example words which occur only once or twice in the dataset."><strong>extremely rare words</strong></span> to reduce computation time?</span>
    <span ng-show="currentStep == 1" class="guidedQuestion">Do you want to import a list of <span uib-tooltip="Each word in the list will be removed from the dataset"><strong>words to remove</strong></span>?</span>
    <span ng-show="currentStep == 2" class="guidedQuestion">Do you want to import a list of <span uib-tooltip="Import your Dictionary with correct words here"><strong>correct words</strong></span>?</span>
    <span ng-show="currentStep == 3" class="guidedQuestion">Do you want to reduce <span uib-tooltip="Like - favourite, favorite, favourites or favorites"><strong>word forms</strong></span> and <span uib-tooltip="Misspelled words like - necesary"><strong>spelling errors</strong></span>?</span>
    <span ng-show="currentStep == 4" class="guidedQuestion">Do you want to find <span uib-tooltip="A tag might be composed of multiple words like - hard rock"><strong>multiword tags</strong></span> which appear very <span uib-tooltip="Frequent multiword tags are as example genres"><strong>often</strong></span>?</span>
    <span ng-show="currentStep == 5" class="guidedQuestion">Do you want to find <span uib-tooltip="A tag might be composed of multiple words like - hard rock"><strong>multiword tags</strong></span> which are <span uib-tooltip="Unique multiword tags are as example artist or band names"><strong>unique</strong></span>?</span>
    <span ng-show="currentStep == 6" class="guidedQuestion">Results</span>
    <span ng-show="currentStep == 7" class="guidedQuestion">Do you want to reduce the total amount of <span uib-tooltip="Filter the tags by there tag quality to remove bad tags"><strong>important words</strong></span> used in your final dataset?</span>
    <span ng-show="currentStep == 8" class="guidedQuestion">Do you want to change those <span uib-tooltip="Remove bad words or translate words or resolve synonyms"><strong>import tags</strong></span> by hand?</span>
    <span ng-show="currentStep == 9" class="guidedQuestion">Results</span>
  </div>
  <br>
  <button ng-if="currentStep > 0" class="btn btn-back" ng-click="ctrl.back()"><i class="fa fa-chevron-left" style="position: relative; top:-2px"></i></button>
  <div class="btn-group" ng-if="currentStep != 6 && currentStep != 9">
    <button id="guided3" ng-click="ctrl.customize()" class="btn secondaryColor" style="width: 150px">Customize <span class="fa fa-gears"></span></button>

    <button id="guided1" ng-if="currentStep < 6" ng-click="ctrl.default()" class="btn primaryColor" style="width: 150px">Use Default <span class="glyphicon glyphicon-play"></span></button>
  </div>
  <br>
  <button ng-if="showStep" ng-click="ctrl.ok()" class="btn primaryColor" style="width: 150px; margin-top: 2px">Apply <span class="glyphicon glyphicon-ok"></span></button>
  <div class="btn-group" ng-if="currentStep == 6 || currentStep == 9">
    <button ng-if="currentStep == 9" ng-click="ctrl.restartFinalize()" class="btn secondaryColor" style="width: 100px">Refine <span class="fa fa-flask"></span></button>
    <button id="result4" ng-if="currentStep < 9" ng-click="ctrl.goToLinked()" class="btn secondaryColor" style="width: 100px">Refine <span class="fa fa-bar-chart"></span></button>
    <button id="result5" ng-click="ctrl.goToAdvanced()" class="btn primaryColor">Advanced Mode <span class="fa fa-list-alt"></span></button>
    <button id="result6" ng-if="currentStep < 9" ng-click="ctrl.ok()" class="btn secondaryColor" style="width: 100px">Finalize <span class="fa fa-flask"></span></button>
  </div>
  <hr>
  <div style="position: relative; height: 100%">
  <div ng-if="currentStep == 0 && ctrl.initRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 3 && ctrl.preRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 4 && ctrl.spellRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 5 && ctrl.spellRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
  <div ng-if="currentStep == 6 && ctrl.compRunning" class="overlayGuided">
    <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
  </div>
    <div ng-if="currentStep == 9 && ctrl.postRunning" class="overlayGuided">
      <div class="overlayInner" spin-rect style="width: 80px; height: 80px"></div>
    </div>
  <div uib-collapse="!showStep" style="text-align: left" class="guidedInner">
    <div ng-show="currentStep == 0">
      <div class="row" ng-controller="PreprocessingfilterCtrl as ctrl" ng-include="'views/preprocessing.filter.html'"></div>
    </div>
    <div ng-show="currentStep == 1">
      <div class="row" ng-controller="PreprocessingimportCtrl as ctrl" ng-include="'views/preprocessing.import.html'"></div>
    </div>
    <div ng-show="currentStep == 2">
      <div class="row" ng-controller="SpellcheckingimportCtrl as ctrl" ng-include="'views/spellchecking.import.html'"></div>
    </div>
    <div ng-show="currentStep == 3">
      <div class="row" ng-controller="SpellcheckingsetCtrl as ctrl" ng-include="'views/spellchecking.set.html'"></div>
    </div>
    <div ng-show="currentStep == 4">
      <div class="row" ng-controller="CompositefrequentCtrl as ctrl" ng-include="'views/composite.frequent.html'"></div>
    </div>
    <div ng-show="currentStep == 5">
      <div class="row" ng-controller="CompositeuniqueCtrl	 as ctrl" ng-include="'views/composite.unique.html'"></div>
    </div>
    <div ng-show="currentStep == 6">
      <div class="row" ng-controller="OverviewCtrl as ctrl" ng-include="'views/overview.html'"></div>
    </div>
    <div ng-show="currentStep == 7">
      <div class="row" ng-controller="PostprocessingimportantCtrl as ctrl" ng-include="'views/postprocessing.important.html'"></div>
    </div>
    <div ng-show="currentStep == 8">
      <div class="row" ng-controller="PostprocessingreplaceCtrl as ctrl" ng-include="'views/postprocessing.replace.html'"></div>
    </div>
    <div ng-show="currentStep == 9">
      <div class="row" ng-controller="OverviewCtrl as ctrl" ng-include="'views/overview.html'"></div>
    </div>
  </div>
    <div id="guided2" ng-if="!showStep">
      <div ng-if="currentStep == 0" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Words - Default</h2>
          <p>We look first at all words separately and polish them. Later, we rebuild all meaningful <span class="descMarker" uib-tooltip="Like: 'Rock n Roll'">multiword tags </span>with the polished words.</p>
          <p>By default no words will be filtered. The removal of words is necessary if your dataset is big and uses a lot of computation time.</p>
          <p>You might wanna start with a small dataset to find good parameters - and run the workflow afterwards with all words.</p>
        </div>
      </div>
      <div ng-if="currentStep == 1" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Blacklist - Default</h2>
          <p>By default <span uib-tooltip="Like: and, or, the, ..."><strong>English stopwords </strong></span> and <span class="descMarker" uib-tooltip="Like: a, b, ... , 1, 2, ..."> single characters</span> will be removed. This reduces computation time and might improve the detection of meaningful single word tags.</p>
          <p>On the other hand stopwords can be an integral part of multiword tags, e.g. "Rock and Roll".</p>
        </div>
      </div>
      <div ng-if="currentStep == 2" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Dictionary - Default</h2>
          <p>By default the <span class="descMarker" uib-tooltip="The most important words for second language learners of English.">General Service List </span>will be imported.</p>
          <p>In small datasets in particular, you might want to import your own list, e.g. genres, to <span uib-tooltip="First the algorithm looks for similar words to the importet dictionary."><strong>improve the quality</strong></span> of the spell correction algorithm.</p>
        </div>
      </div>
      <div ng-if="currentStep == 3" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Spell Correction - Default</h2>
          <p>By default, words with a <span uib-tooltip="Is computed from the intial weights"><strong>word quality</strong></span> above <strong>0.7 </strong> will be treated as correct and words with a similarity above <strong> 0.7</strong> will be replaced by the word with the higher quality.</p>
        </div>
      </div>
      <div ng-if="currentStep == 4" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Frequent Multiword Tags - Default</h2>
          <p>By default a group strength of <strong>0.35 </strong>is used.</p>
          <p>1 Means this group appeared most often and 0 it appeared never.</p>
        </div>
      </div>
      <div ng-if="currentStep == 5" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Unique Multiword Tags - Default</h2>
          <p>By default a group strength of <strong>0.7 </strong>will be used.</p>
          <p>1 Means this two words occur very often together an are less separated and 0 means the words are never together and often separated</p>
        </div>
      </div>
      <div ng-if="currentStep == 7" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Final Filtering</h2>
          <p>Select a set of <strong>important tags</strong>.</p>
          <p>Only the select tags <span class="descMarker" uib-tooltip="There is a option in the advanced mode to use all tags at the end but only the selected ones for the following salvage step.">remain</span> in the dataset.</p>
          <p>The following step searches in the tags below the selected threshold for the words above or equal the threshold.</p>
        </div>
      </div>
      <div ng-if="currentStep == 8" class="guidedExplanation">
        <div class="well well-large center" style="width: 600px">
          <h2>Salvager</h2>
          <p>You can change or remove tags from the selected tags. You might want to remove non-informative words or fix synonyms by hand.</p>
          <p>Afterwards the salvager searches for the updated important tags in the not important tags. This increases the total dataset coverage.</p>
        </div>
      </div>
    </div>
  </div>
</div>

{'fraction_non_alphanumeric': 0.15833073531376654, 'fraction_numerical': 0.014474680443593284, 'mean_word_length': 4.0469768487300515, 'pattern_counts': {'":': 0, '<': 595, '<?xml version=': 0, '>': 591, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2809337', 'n_tokens_mistral': 7063, 'n_tokens_neox': 6659, 'n_words': 2033}