<commit_before>---
layout: page 
title: Named Entity Recognition against Custom Dataset
dropdown: Tutorials
pos: 3.1
---

<h1>Still draft..</h1>

<h1>Named Entity Recognition against Custom Dataset</h1>
<div>
    <h2>Introduction</h2>
    <p>Most named entity recognition tools (NER) perform linking of the entities occurring in the text with only one dataset provided by the NER system.
    However, very often a user would like to match (link) the entities occurring in the document with a proprietary domain specific dataset.
    In this tutorial you can learn how to perform named entity linking against a custom dataset using the e-Entity sevices.</p>
</div>

<div>
    <h2>Pre-requisites</h2>
    <p>
    	<ol>
    		<li>A custom dataset provided in RDF containing entities and labels for the entities.</li>
    		<li>A document which will be used for processing.</li>
    	</ol>
    </p>
</div>

<script type="text/javascript">
	
    function sendDatasetCreateRequest() {

	$.ajax({
	    url: fremeApiUrl + "/e-entity/freme-ner/datasets?"
	    +"informat="+$("#informat-dataset").val()
	    +"&name="+$("#dataset-name").val()
	    +"&description="+$("#dataset-description").val()
	    +"&language=en"
	    +"&outformat=json",
            type: "POST",
	    dataType: "text",
	    contentType: 'text/plain',
            data: $("#input-dataset").val()
        }).done(function(result){
    	    $("#dataset-creation-response").text(result);
	});
       };

    function sendTextProcessingRequest() {

	$.ajax({
	    url: fremeApiUrl + "/e-entity/freme-ner/documents",
            type: "POST",
	    dataType: "text",
            data: {
                'informat': encodeURIComponent($("#informat-text").val()),
                'outformat': encodeURIComponent($("#outformat-text").val()),
                'language': encodeURIComponent($("#text-language").val()),
                'dataset': encodeURIComponent($("#target-dataset").val()),
                //'input': encodeURIComponent($("#input-text").val())
                'input': $("#input-text").val()
            }
        }).done(function(result){
    	    $("#ner-results").text(result);
	}).fail(function(result) {
    	    $("#ner-results").text(JSON.stringify(result));
	});
       };
       
</script>

<div>
    <h2>Step 1: Prepare your dataset</h2>
    <p>First we need to prepare our custom dataset.
    The dataset should be provided in RDF and contain list of entities names for the entities.
    Bellow is small example of such dataset in the N-triples format where in each record, the subject (first element) is a URI identifier for the entity, the predicate (second element) is the type of the information we describe (preferred or alternative label for the entity), and the object (last element) is the actual name for the entity.</p>
    <!-- HTML generated using hilite.me --><div style="background: #ffffff; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">&lt;http://www.freme-projects.eu/dataset/people/Milan<span style="color: #007020">_</span>Dojchinovski&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Milan<span style="color: #007020">_</span>Dojchinovski&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Milan&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Sebastian<span style="color: #007020">_</span>Hellmann&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Sebastian<span style="color: #007020">_</span>Hellmann&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Sebastian&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Felix<span style="color: #007020">_</span>Sasaki&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Felix<span style="color: #007020">_</span>Sasaki&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Felix&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Jan<span style="color: #007020">_</span>Nehring&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .<commit_msg>NER with custom dataset tutorial<commit_after>---
layout: page 
title: Named Entity Recognition against Custom Dataset
dropdown: Tutorials
pos: 3.1
---

<h1>Still draft..</h1>

<h1>Named Entity Recognition against Custom Dataset</h1>
<div>
    <h2>Introduction</h2>
    <p>Most named entity recognition tools (NER) perform linking of the entities occurring in the text with only one dataset provided by the NER system.
    However, very often a user would like to match (link) the entities occurring in the document with a proprietary domain specific dataset.
    In this tutorial you can learn how to perform named entity linking against a custom dataset using the e-Entity sevices.</p>
</div>

<div>
    <h2>Pre-requisites</h2>
    <p>
    	<ol>
    		<li>A custom dataset provided in RDF containing entities and labels for the entities.</li>
    		<li>A document which will be used for processing.</li>
    	</ol>
    </p>
</div>

<script type="text/javascript">
	
    function sendDatasetCreateRequest() {

	$.ajax({
	    url: fremeApiUrl + "/e-entity/freme-ner/datasets?"
	    +"informat="+$("#informat-dataset").val()
	    +"&name="+$("#dataset-name").val()
	    +"&description="+$("#dataset-description").val()
	    +"&language=en"
	    +"&outformat=json",
            type: "POST",
	    dataType: "text",
	    contentType: 'text/plain',
            data: $("#input-dataset").val()
        }).done(function(result){
    	    $("#dataset-creation-response").text(result);
	});
       };

    function sendTextProcessingRequest() {

	$.ajax({
	    url: fremeApiUrl + "/e-entity/freme-ner/documents",
            type: "POST",
	    dataType: "text",
	    contentType: '',
            data: {
                'informat': encodeURIComponent($("#informat-text").val()),
                'outformat': encodeURIComponent($("#outformat-text").val()),
                'language': encodeURIComponent($("#text-language").val()),
                'dataset': encodeURIComponent($("#target-dataset").val()),
                //'input': encodeURIComponent($("#input-text").val())
                'input': $("#input-text").val()
            }
        }).done(function(result){
    	    $("#ner-results").text(result);
	}).fail(function(result) {
    	    $("#ner-results").text(JSON.stringify(result));
	});
       };
       
</script>

<div>
    <h2>Step 1: Prepare your dataset</h2>
    <p>First we need to prepare our custom dataset.
    The dataset should be provided in RDF and contain list of entities names for the entities.
    Bellow is small example of such dataset in the N-triples format where in each record, the subject (first element) is a URI identifier for the entity, the predicate (second element) is the type of the information we describe (preferred or alternative label for the entity), and the object (last element) is the actual name for the entity.</p>
    <!-- HTML generated using hilite.me --><div style="background: #ffffff; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%">&lt;http://www.freme-projects.eu/dataset/people/Milan<span style="color: #007020">_</span>Dojchinovski&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Milan<span style="color: #007020">_</span>Dojchinovski&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Milan&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Sebastian<span style="color: #007020">_</span>Hellmann&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Sebastian<span style="color: #007020">_</span>Hellmann&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Sebastian&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Felix<span style="color: #007020">_</span>Sasaki&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Felix<span style="color: #007020">_</span>Sasaki&gt; &lt;http://www.w3.org/2004/02/skos/core#altLabel&gt; &quot;Felix&quot; .
&lt;http://www.freme-projects.eu/dataset/people/Jan<span style="color: #007020">_</span>Nehring&gt; &lt;http://www.w3.org/2004/02/skos/core#prefLabel&gt; &quot;<NAME>&quot; .
{'fraction_non_alphanumeric': 0.17538568611529984, 'fraction_numerical': 0.026447047906275375, 'mean_word_length': 3.696078431372549, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 103, 'https://': 0, 'lorem ipsum': 0, 'www.': 28, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4115915', 'n_tokens_mistral': 3018, 'n_tokens_neox': 2822, 'n_words': 621}