<commit_before>ckanext-stadtzh-harvest
=======================

Harvester for the City of Zurich.
The data is loaded from so called "dropzones", which are mounted network drives.
Each folder in a dropzone is considered a dataset.

## Content

* [Configuration](#configuration)
* [Metadata](#metadata)
	* [meta.xml](#metaxml)
	* [link.xml](#linkxml)


## Configuration

The harvest configuration must be provided as a valid JSON string.

Example:

```json
{
    "data_path": "/home/liip/dropzones/GEO",
    "metadata_dir": "geo-metadata",
    "metafile_dir": "DEFAULT",
    "update_datasets": false,
    "update_date_last_modified": true
}
```

### `data_path`

The path to the dropzone

### `metadata_dir`

The path where the diff files are stored, this is used to compare the metadata from the previous days harvest job.

### `metafile_dir`

The name of the directory where the `meta.xml` is located.
The GEO dropzone has a subdirectory for the meta.xml, all other dropzones should provide an empty string here.

### `update_datasets`

Boolean flag (true/false) to determine if this harvester should update existing datasets or not.
If the flag is `false` no updates will be performed, only new datasets will be added.

### `update_date_last_modified`

Boolean flag (true/false) to determine if the field `date_last_modified` of a dataset should be updated by the harvester or not.
If the flag is `true` the date will be updated if the content of any resource of a dataset has changed.

## Metadata
Each dataset consists of a folder containing a `meta.xml` (required!) and an arbitrary number of resources.

You can find examples for `meta.xml` and `link.xml` files in the [`fixtures` directory of this repository](https://github.com/opendatazurich/ckanext-stadtzh-harvest/tree/master/ckanext/stadtzhharvest/tests/fixtures).

### meta.xml

Folder structure:
```
DWH/bev_zuz_jahr_quartier
├── 1993-2012_bev_zuz_jahr_quartier.csv
├── bev_zuz_jahr_quartier.csv
└── meta.xml
```

#### `titel`

* **Data type**: String
* **Cardinality**: 1
* **Description**: Title of this dataset
* **Values**: any literal
* **Example**:
```xml
<titel>Alterswohnung</title>
```
<commit_msg>add reference to xsd-Files<commit_after>ckanext-stadtzh-harvest
=======================

Harvester for the City of Zurich.
The data is loaded from so called "dropzones", which are mounted network drives.
Each folder in a dropzone is considered a dataset.

## Content

* [Configuration](#configuration)
* [Metadata](#metadata)
	* [meta.xml](#metaxml)
	* [link.xml](#linkxml)


## Configuration

The harvest configuration must be provided as a valid JSON string.

Example:

```json
{
    "data_path": "/home/liip/dropzones/GEO",
    "metadata_dir": "geo-metadata",
    "metafile_dir": "DEFAULT",
    "update_datasets": false,
    "update_date_last_modified": true
}
```

### `data_path`

The path to the dropzone

### `metadata_dir`

The path where the diff files are stored, this is used to compare the metadata from the previous days harvest job.

### `metafile_dir`

The name of the directory where the `meta.xml` is located.
The GEO dropzone has a subdirectory for the meta.xml, all other dropzones should provide an empty string here.

### `update_datasets`

Boolean flag (true/false) to determine if this harvester should update existing datasets or not.
If the flag is `false` no updates will be performed, only new datasets will be added.

### `update_date_last_modified`

Boolean flag (true/false) to determine if the field `date_last_modified` of a dataset should be updated by the harvester or not.
If the flag is `true` the date will be updated if the content of any resource of a dataset has changed.

## Metadata
Each dataset consists of a folder containing a `meta.xml` (required!) and an arbitrary number of resources.

You can find examples for `meta.xml` and `link.xml` files in the [`fixtures` directory of this repository](https://github.com/opendatazurich/ckanext-stadtzh-harvest/tree/master/ckanext/stadtzhharvest/tests/fixtures).

You can use the [`meta.xsd`](https://github.com/opendatazurich/ckanext-stadtzh-harvest/blob/master/meta.xsd) and [`link.xsd`](https://github.com/opendatazurich/ckanext-stadtzh-harvest/blob/master/link.xsd) for validation ( e.g. using a free online [XML Validator](https://www.liquid-technologies.com/online-xsd-validator) )

### meta.xml

Folder structure:
```
DWH/bev_zuz_jahr_quartier
├── 1993-2012_bev_zuz_jahr_quartier.csv
├── bev_zuz_jahr_quartier.csv
└── meta.xml
```

#### `titel`

* **Data type**: String
* **Cardinality**: 1
* **Description**: Title of this dataset
* **Values**: any literal
* **Example**:
```xml
<titel>Alterswohnung</title>
```

{'fraction_non_alphanumeric': 0.12441214194100043, 'fraction_numerical': 0.003847798204360838, 'mean_word_length': 4.878140703517588, 'pattern_counts': {'":': 10, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 24}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '433906', 'n_tokens_mistral': 1505, 'n_tokens_neox': 1450, 'n_words': 515}