<commit_before># Generation of metadata
This document describes how formated metadata records can be generated from given raw information. 

## Environment
Ubuntu 14.04 server

## Prerequisites

### 1. Python and the convert script
* Python 2.7 or later
#### 1.1. The script mdmanger.py
The python script ```mdmanger.py (mode ‘g’)``` is used for this training module and comes already with the training package.
#### 1.2
The needed pyton module *dublincore* can be installed by
```sh
pip install dublincore
```
> Note : If you get an error like
```sh
Requirement already satisfied (use --upgrade to upgrade): dublincore in /home/xxx/anaconda2/lib/python2.7/site-packages
```
> you have to add this path to your Python path by
```sh
export PYTHONPATH=$PYTHONPATH:/home/k204019/anaconda2/lib/python2.7/site-packages/
```

### 2. Some samples of raw data
Some examples of 'raw metadata', i.e. comma and tabulator seperated lists,are given in the directory *samples/RAW_data*.
```sh
ls samples/RAW_data
annotations.tsv  sample.csv  smallData.tsv
```
At the moment this part ist tested with the comma separated list in ```sample.csv```.

## Usage

### 1. Generation of DublinCore XML files

Please check firts the usage of the script by :
```sh
./mdmanager.py -h
```
For this part we will only use the `Single Mode Options`
```sh
Use these options if you want to ingest from only ONE source.

--source=PATH, -s PATH  A URL to .xml files which you want to harvest
--verb=STRING           Verbs or requests defined in OAI-PMH, can be
                        ListRecords (default) or ListIdentifers here
--mdsubset=STRING       Subset of harvested meta data
--mdprefix=STRING       Prefix of harvested meta data
--target_mdschema=STRING
                        Meta data schema of the target
```

In the following we will use this script to convert comma or tabular seperated list of values to valid XML files formated in the [DublinCore](http://dublincore.org/) metadata schema. The usage of the several options mentioned above will explained in the following.

#### 1.2. Creation of a mapfile
In a first step we define the mapping of each field in the header line of the sample file
```sh
$ head -1 samples/RAW_data/sample.csv 
Common name,Scientific name,Location,Temperament,Diet,Water,Size,Region of the Aquarium,Breeding
```
to one of the DublinCore elements (see ```sh mapfiles/dcelements.txt```) or of the DublinCore terms (see ```sh mapfiles/dcterms.txt```).

You are free by assigning the original fields to one of the dublincore fields. The only mandatory fields to set are ```dc:title``` and ```dc:identifier```, whereby the value of the latter is used to build the filename of the generated XML record. 

Beside the mode (`--mode h`) and the source (`-s FILE`), i.e. here the path to the csv file, the following options are mandatory :
- ```-c COMMUNITY``` : a free choosen project or community name. Here and in the following modules we call our project *fishproject*.
- ```--mdprefix MDFORMAT``` : This is the target metadata schema, we choose here *oai_dc*, i.e. teh Dublincore metadata schema.
- ```--mdsubset MDSUBSET``` : The subset that deteermines the sub directory the generated XML files reside.
- ```--verb DELIMITER``` : specifies the delimiter by which the sample data are seperated, in our example *comma*.

From the first two parameters the name for the mapfile (as well a cvs file) is built as ```mapfiles/<community>-<mdprefix>.csv``` and from the first three the output path is combind to ```oaidata/<community>-<mdprefix>/<mdsubset>_1```.  

By calling the script with the csv file as source for the first time, there will be now mapfile available. So you are asked for each original header field to enter a valid DublinCore element or term :

```sh
./mdmanager.py --mode g -s samples/RAW_data/sample.csv -c fishproject --mdprefix oai_dc --mdsubset sample --verb comma

Target field for Common name : dc:identifier
Target field for Scientific name : dc:title
Target field for Location : dcterms:spatial
Target field for Temperament : dc:type
Target field for Diet : dcterms:requires
Target field for Water : dcterms:medium
Target field for Size : dcterms:extent
Target field for Region of the Aquarium : dcterms:provenance
Target field for Breeding : dcterms:created
```

This results not only in the newly generated mapping file ```sh mapfiles/mymapfile.cvs``` but as well the mapping is excecuted and the associated XML files are written to the output directory ```sh outdata```.

#### Excercise 2.1: Repeat the generation of mapfile and do your own mapping
#### Excercise 2.2: Check the mapping rules in the file you entered with the option ```sh -c ``` (in teh example above it's the file ```sh .
#### Excercise 2.3: Check the results, i.e. the generated XML files in the output directory,e.g. with the above mapping oen of teh resulted files looks as follows:

Optionally an output directory for the generated XML files can be indicated by option ```sh -o OUTPATH```.

```sh  
less myout/Compressiceps-DC.xml
<?xml version="1.0"?>

<metadata
    xmlns="http://example.org/myapp/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://purl.org/dc/elements/1.1/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dcterms="http://purl.org/dc/terms/">

        <dc:identifier>Compressiceps</dc:identifier>
        <dc:title>Haplochromis compressiceps</dc:title>
        <dc:type>Territorial</dc:type>
        <dcterms:created>Hard</dcterms:created>
        <dcterms:extent>5 inches</dcterms:extent>
        <dcterms:medium>PH 7.0 - 8.0, Temp. 73 - 77 F</dcterms:medium>
        <dcterms:provenance>Bottom</dcterms:provenance>
        <dcterms:requires>Omnivore</dcterms:requires>
        <dcterms:spatial>Lake Tanganyika</dcterms:spatial>
</metadata>
``` 

> Note : Here starts the issue of semantic mapping and 
> homogenisation of a research specific vocabulary to a common, 
> restricted vocabulary or schema as DublinCore. I.e. the mapping above is quite "formal", e.g. above *`Diet`* is mapped to *`dcterms:requires`*, but likely the intention of the DublinCore consortium by introducing the field *requires* was not in the sense of `a plant requires a specific diet`. So, if we want to cover these specific biological properties we have to add another extented shema, which covers the property 'Diet'. How this can be done is explained in the next section. 

To change the mapping rules to your needs you can instead - running the script repeatately with option ```-c``` - as well edit the mapfile, whereby the seperator ´´´sh >´´´ is used to seperate the original field name and the target Dublincore element, as shown here :

```sh 
$ cat mapfiles/mymapfile.cvs 
Common name>dc:identifier
Scientific name>dc:title
Location>dcterms:spatial
Temperament> dc:type
Diet>dcterms:requires
Water>dcterms:medium
Size> dcterms:extent
Region of the Aquarium>dcterms:provenance
Breeding>dcterms:created
```
 
### 2. Subfollowing converting
With an existing (and reviewed) mapfile the script can be repeatately called with the option ```sh -m MAPFILE```.

```sh
./md-convert.py -m mapfiles/myMapFile.cvs [ -o myOutPath ] samples/RAW_data/sample.csv
```

### 3. Check the results

The resulting XML files in should now compatible to the DublinCore metadata format.

Excercise 3.1: You can check this e.g. by one of the available XML validators, e.g. ```sh http://www.xmlvalidation.com/```.
<commit_msg>Typos and other improvements<commit_after># Generation of metadata
This document describes how formated metadata records can be generated from given raw information. 

## Environment
Ubuntu 14.04 server

## Prerequisites

### 1. Python and the convert script
* Python 2.7 or later

#### 1.1. The script mdmanger.py
The python script ```mdmanger.py (mode ‘g’)``` is used for this training module and comes already with the training package.

#### 1.2
The needed pyton module *dublincore* can be installed by
```sh
pip install dublincore
```
> Note : If you get an error like
```sh
Requirement already satisfied (use --upgrade to upgrade): dublincore in /home/xxx/anaconda2/lib/python2.7/site-packages
```
> you have to add this path to your Python path by
```sh
export PYTHONPATH=$PYTHONPATH:/home/k204019/anaconda2/lib/python2.7/site-packages/
```

### 2. Some samples of raw data
Some examples of 'raw metadata', i.e. comma and tabulator seperated lists,are given in the directory *samples/RAW_data*.
```sh
ls samples/RAW_data
annotations.tsv  sample.csv  smallData.tsv
```
At the moment this part ist tested with the comma separated list in ```sample.csv```.

## Usage

### 1. Generation of DublinCore XML files

Please check first the usage of the script by :
```sh
./mdmanager.py -h
```
For this part we will only use the `Single Mode Options`
```sh
Use these options if you want to ingest from only ONE source.

--source=PATH, -s PATH  A URL to .xml files which you want to harvest
--verb=STRING           Verbs or requests defined in OAI-PMH, can be
                        ListRecords (default) or ListIdentifers here
--mdsubset=STRING       Subset of harvested meta data
--mdprefix=STRING       Prefix of harvested meta data
--target_mdschema=STRING
                        Meta data schema of the target
```

In the following we show how you can convert comma or tabular seperated lists of values to valid XML files formated in the [DublinCore](http://dublincore.org/) metadata schema. The usage of the several options mentioned above will explained in the following.

#### 1.2. Creation of a mapfile
In a first step we define the mapping of each field in the header line of the sample file
```sh
$ head -1 samples/RAW_data/sample.csv 
Common name,Scientific name,Location,Temperament,Diet,Water,Size,Region of the Aquarium,Breeding
```
to one of the DublinCore elements (see ```sh mapfiles/dcelements.txt```) or the DublinCore terms (see ```sh mapfiles/dcterms.txt```).

You are free by assigning the original fields to any one of the dublincore fields. The only mandatory fields to set are ```dc:title``` and ```dc:identifier```, whereby the value of the latter is used to build the filename of the generated XML record. 

Beside the mode (`--mode g`, standing for genaration) and the source (`-s FILE`), i.e. the path to the csv file, the following options are mandatory :
- ```-c COMMUNITY``` : a free choosen project or community name. Here and in the following modules we call our project *fishproject*.
- ```--mdprefix MDFORMAT``` : This is the target metadata schema, we choose here *oai_dc*, i.e. the Dublincore metadata schema.
- ```--mdsubset MDSUBSET``` : The subset that determines the sub directory the generated XML files reside.
- ```--verb DELIMITER``` : specifies the delimiter by which the sample data are seperated, in our example *comma*.

From the first two parameters the name for the mapfile (as well a cvs file) is built as ```mapfiles/<community>-<mdprefix>.csv``` and from the first three the output path is combind to ```oaidata/<community>-<mdprefix>/<mdsubset>_1```.  

By calling the script with the csv file as source for the first time, there will be no mapfile available. So you are asked for each original header field to enter a valid DublinCore element or term :

```sh
./mdmanager.py --mode g -s samples/RAW_data/sample.csv -c fishproject --mdprefix oai_dc --mdsubset sample --verb comma

Target field for Common name : dc:identifier
Target field for Scientific name : dc:title
Target field for Location : dcterms:spatial
Target field for Temperament : dc:type
Target field for Diet : dcterms:requires
Target field for Water : dcterms:medium
Target field for Size : dcterms:extent
Target field for Region of the Aquarium : dcterms:provenance
Target field for Breeding : dcterms:created
```

This results not only in the newly generated mapping file ```fishproject-oai_dc.csv```, but as well the mapping is excecuted and the associated XML files are written to the output directory ```oaidata/fishproject-oai_dc/sample_1/xml/```.

#### Excercise 2.1: Repeat the generation and do your own mapping by adapting the mapping rules in teh mapfile to your needs. Hereby the seperator ´´´>´´´ is used to seperate the original field name and the target Dublincore element or term :
```sh
$ vi mapfiles/fishproject-oai_dc.csv
ommon name>dc:identifier
Scientific name>dc:title
Location>dcterms:spatial
Temperament> dc:type
Diet>dcterms:requires
Water>
Size> dcterms:extent
Region of the Aquarium>dcterms:provenance
Breeding>dcterms:created
```
 
#### Excercise 2.2: Check the results, i.e. the generated XML files in the output directory,e.g. with the above mapping one of the resulted files looks as follows:

```sh  
less myout/Compressiceps-DC.xml
<?xml version="1.0"?>

<metadata
    xmlns="http://example.org/myapp/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://purl.org/dc/elements/1.1/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dcterms="http://purl.org/dc/terms/">

        <dc:identifier>Compressiceps</dc:identifier>
        <dc:title>Haplochromis compressiceps</dc:title>
        <dc:type>Territorial</dc:type>
        <dcterms:created>Hard</dcterms:created>
        <dcterms:extent>5 inches</dcterms:extent>
        <dcterms:provenance>Bottom</dcterms:provenance>
        <dcterms:requires>Omnivore</dcterms:requires>
        <dcterms:spatial>Lake Tanganyika</dcterms:spatial>
</metadata>
``` 
You can check this e.g. by one of the available XML validators, e.g. ```sh http://www.xmlvalidation.com/```.

> Note : Here starts the issue of semantic mapping and 
> homogenisation of a research specific vocabulary to a common, 
> restricted vocabulary or schema as DublinCore. I.e. the mapping above is quite "formal", e.g. `Diet` is mapped to `dcterms:requires`, but likely the intention of the DublinCore consortium by introducing the field `requires` was not in the sense of `a plant requires a specific diet`. So, if we want to cover these specific biological properties we have to add another extented shema, which covers the property `Diet`. 



{'fraction_non_alphanumeric': 0.09787535410764872, 'fraction_numerical': 0.007436260623229462, 'mean_word_length': 4.157414170927685, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 2, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 17}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '483065', 'n_tokens_mistral': 4391, 'n_tokens_neox': 4166, 'n_words': 1777}