<commit_before><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<chapter id="tools.postagger">
<title>Part-of-Speech Tagger</title>
	<section id="tools.postagger.tagging">
		<title>Tagging</title>
		<para>
		The Part of Speech Tagger marks tokens with their corresponding word type
		based on the token itself and the context of the token. A token can have
		multiple pos tags depending on the token and the context. The OpenNLP POS Tagger
		uses a probability model to guess the correct pos tag out of the tag set.
		To limit the possible tags for a token a tag dictionary can be used which increases
		the tagging and runtime performance of the tagger.
		</para>
			<section id="tools.postagger.tagging.cmdline">
		<title>POS Tagger Tool</title>
		<para>
		The easiest way to try out the POS Tagger is the command line tool. The tool is only intended for demonstration and testing.
		Download the english maxent pos model and start the POS Tagger Tool with this command:
		<screen>
			<![CDATA[
$ bin/opennlp POSTagger en-pos-maxent.bin]]>
		 </screen>
		The POS Tagger now reads a tokenized sentence per line from stdin.
		Copy these two sentences to the console:
		<programlisting>
			<![CDATA[
<NAME> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .]]>
		 </programlisting>
		 the POS Tagger will now echo the sentences with pos tags to the console:
		<programlisting>
			<![CDATA[
Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
Mr._NNP Vinken_NNP is_VBZ chairman_NN of_IN Elsevier_NNP N.V._NNP ,_, the_DT Dutch_NNP publishing_VBG group_NN]]>
		 </programlisting> 
		 The tag set used by the english pos model is the Penn Treebank tag set. See the link below for a description of the tags.
		</para>
		</section>
	</section>
</chapter><commit_msg>OPENNLP-64 Added a section about the tagger API.<commit_after><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<chapter id="tools.postagger">
<title>Part-of-Speech Tagger</title>
	<section id="tools.postagger.tagging">
		<title>Tagging</title>
		<para>
		The Part of Speech Tagger marks tokens with their corresponding word type
		based on the token itself and the context of the token. A token might have
		multiple pos tags depending on the token and the context. The OpenNLP POS Tagger
		uses a probability model to predict the correct pos tag out of the tag set.
		To limit the possible tags for a token a tag dictionary can be used which increases
		the tagging and runtime performance of the tagger.
		</para>
			<section id="tools.postagger.tagging.cmdline">
		<title>POS Tagger Tool</title>
		<para>
		The easiest way to try out the POS Tagger is the command line tool. The tool is only intended for demonstration and testing.
		Download the english maxent pos model and start the POS Tagger Tool with this command:
		<screen>
			<![CDATA[
$ bin/opennlp POSTagger en-pos-maxent.bin]]>
		 </screen>
		The POS Tagger now reads a tokenized sentence per line from stdin.
		Copy these two sentences to the console:
		<programlisting>
			<![CDATA[
<NAME> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .]]>
		 </programlisting>
		 the POS Tagger will now echo the sentences with pos tags to the console:
		<programlisting>
			<![CDATA[
Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
Mr._NNP Vinken_NNP is_VBZ chairman_NN of_IN Elsevier_NNP N.V._NNP ,_, the_DT Dutch_NNP publishing_VBG group_NN]]>
		 </programlisting> 
		 The tag set used by the english pos model is the Penn Treebank tag set. See the link below for a description of the tags.
		</para>
      </section>
      
		<section id="tools.postagger.tagging.api">
		<title>POS Tagger API</title>
		<para>
		    The POS Tagger can be embedded into an application via its API.
			First the pos model must be loaded into memory from disk or an other source.
			In the sample below its loaded from disk.
			<programlisting language="java">
				<![CDATA[
InputStream modelIn = null;

try {
  modelIn = new FileInputStream("en-pos-maxent.bin");
  POSModel model = new POSModel(modelIn);
}
catch (IOException e) {
  // Model loading failed, handle the error
  e.printStackTrace();
}
finally {
  if (modelIn != null) {
    try {
      modelIn.close();
    }
    catch (IOException e) {
    }
  }
}]]>
			</programlisting>
			After the model is loaded the POSTaggerME can be instantiated.
			<programlisting language="java">
				<![CDATA[
POSTaggerME tagger = new POSTaggerME(model);]]>
			</programlisting>
			The POS Tagger instance is now ready to tag data. It expects a tokenized sentence
			as input, which is represented as a String array, each String object in the array
			is one token.
	   </para>
	   <para>
	   The following code shows how to determine the most likely pos tag sequence for a sentence.
	   	<programlisting language="java">
		  <![CDATA[
String sent[] = new String[]{"Most", "large", "cities", "in", "the", "US", "had",
                             "morning", "and", "afternoon", "newspapers", "."};		  
String tags[] = tagger.tag(sent);]]>
			</programlisting>
			The tags array contains one part-of-speech tag for each token in the input array. The corresponding
			tag can be found at the same index as the token has in the input array.
			The confidence scores for the returned tags can be easily retrieved from
			a POSTaggerME with the following method call:
				   	<programlisting language="java">
		  <![CDATA[
double probs[] = tagger.probs();]]>
			</programlisting>
			The call to probs is stateful and will always return the probabilities of the last
			tagged sentence. The probs method should only be called when the tag method
			was called before, otherwise the behavior is undefined.
			</para>
			<para>
			Some applications need to retrieve the n-best pos tag sequences and not
			only the best sequence.
			The topKSequences method is capable of returning the top sequences.
			It can be called in a similar way as tag.
			<programlisting language="java">
		  <![CDATA[
Sequence topSequences[] = tagger.topKSequences(sent);]]>
			</programlisting>	
			Each Sequence object contains one sequence. The sequence can be retrieved
			via Sequence.getOutcomes() which returns a tags array 
			and Sequence.getProbs() returns the probability array for this sequence.
	  		 </para>
	</section>
	</section>
</chapter>
{'fraction_non_alphanumeric': 0.08046661111772407, 'fraction_numerical': 0.004761337935960005, 'mean_word_length': 3.7792946530147895, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 2, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7375390', 'n_tokens_mistral': 2669, 'n_tokens_neox': 2359, 'n_words': 1101}