<commit_before># meddo

Data source for Wikipedia maps from OSM data

## Install

This style requires an osm2pgsql database loaded with [https://github.com/ClearTables/ClearTables](https://github.com/ClearTables/ClearTables) and ocean data.

### Requirements

* ClearTables v0.0.8
  The version required is subject to **rapid** change during development
* Mapnik 3.0.0 or later
* Software that can interpret YAML style definitions like Kosmtik or Mapbox Studio Classic
* osm2pgsql 0.90.1 or later with Lua support. Early C++ versions > 0.86.0 may still work with some bugs or missing data.
* PostgreSQL 9.3 or later. 9.4 or later is recommended as earlier versions are not adequately tested with the style.
* PostGIS 2.0 or later. 2.3 or later is recommended as earlier versions are not adequately tested with the style.
* Python 3.4 or later

### Load the data with ClearTables

See the [ClearTables documentation for details](https://github.com/ClearTables/ClearTables#usage) and load into the database `ct`, with the command-line option `-G`

```sh
git clone -b v0.0.8 git://github.com/ClearTables/ClearTables.git
pushd ClearTables
make
createdb ct
psql -d ct -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
make # Create cleartables files
cat sql/types/*.sql | psql -1Xq -d ct # Load SQL types
osm2pgsql -G -d ct --number-processes 2 --output multi --style cleartables.json ~/path/to/extract # Load OSM data
cat sql/post/*.sql | psql -1Xq -d ct # Add post-import SQL
popd
```

Other osm2pgsql flags for performance or updates can be added, and will be necessary for large imports. See the osm2pgsql documentation for more details.
Flags that might be needed include
- `--slim`
- `--cache`
- `--flat-nodes`

Slim mode is not required by this style, so ``--slim --drop`` can be safely used if updates are not required.

If PostgreSQL [`max_connections`](http://www.postgresql.org/docs/9.3/static/runtime-config-connection.html#RUNTIME-CONFIG-CONNECTION-SETTINGS)
is increased from the default, `--number-processes` can be increased. If `--number-processes` is omitted, osm2pgsql will
attempt to use as many processes as hardware threads.

### Load coastline data

Meddo uses data from [OSMCoastline](https://github.com/osmcode/osmcoastline), hosted on [OpenStreetMapData](http://openstreetmapdata.com/), and from [Natural Earth](http://www.naturalearthdata.com/). The data used is documented in full in [`external-data.yml`](external-data.yml)

```sh
./get-external-data.py
```

It should be re-run periodically, and the command options can be obtained with `./get-external-data.py -h`

### Load admin data
Meddo uses data generated by [OSMBorder](https://github.com/pnorman/osmborder) for administrative boundaries.

Generate the borders with OSMBorder then run the SQL

```sql
CREATE TABLE osmborder_lines (
  osm_id bigint,
  admin_level int,
  disputed bool,
  maritime bool,
  way Geometry(LineString, 3857));

\copy osmborder_lines FROM osmborder_lines.csv

CREATE INDEX osmborder_lines_way_idx ON osmborder_lines USING gist (way) WITH (fillfactor=100);
CLUSTER osmborder_lines USING osmborder_lines_way_idx;
CREATE INDEX osmborder_lines_way_low_idx ON osmborder_lines USING gist (way) WITH (fillfactor=100) WHERE admin_level <= 4;
```

### Install required functions

Meddo requires some standard stylesheet-independent functions

```sh
psql -d ct -f functions.sql
```

## Usage

### Development

A suitable design program like Kosmtik or Mapbox Studio Classic is needed. If Kosmtik is installed, `kosmtik serve data.yml` will start Kosmtik, and the Data Inspector can be used. For Mapbox Studio, the entire repository is a tm2source project.

### Production

It might be necessary to compile the project to Mapnik XML for production, which can be done in many ways. One way is `kosmtik export data.yml --format xml --output meddo.xml`

## Schema

*TBD.*

## Why "meddo"?

[Meddo](https://en.wikipedia.org/wiki/Marshall_Islands_stick_chart#Meddo_charts) is a type of [Polynesian stick chart](https://en.wikipedia.org/wiki/Marshall_Islands_stick_chart) which contains only a section of the island chain, and meddo creates vector tiles from a larger database. It's also based on the same language as "wiki".

## License

The code is licensed under the [MIT License](LICENSE). If used as directed, use [ODbL licensed OpenStreetMap data](https://www.openstreetmap.org/copyright).
<commit_msg>Add extensions needed for name matching<commit_after># meddo

Data source for Wikipedia maps from OSM data

## Install

This style requires an osm2pgsql database loaded with [https://github.com/ClearTables/ClearTables](https://github.com/ClearTables/ClearTables) and ocean data.

### Requirements

* ClearTables v0.0.8
  The version required is subject to **rapid** change during development
* Mapnik 3.0.0 or later
* Software that can interpret YAML style definitions like Kosmtik or Mapbox Studio Classic
* osm2pgsql 0.90.1 or later with Lua support. Early C++ versions > 0.86.0 may still work with some bugs or missing data.
* PostgreSQL 9.3 or later. 9.4 or later is recommended as earlier versions are not adequately tested with the style.
* PostGIS 2.0 or later. 2.3 or later is recommended as earlier versions are not adequately tested with the style.
* Python 3.4 or later

### Load the data with ClearTables

See the [ClearTables documentation for details](https://github.com/ClearTables/ClearTables#usage) and load into the database `ct`, with the command-line option `-G`

```sh
git clone -b v0.0.8 git://github.com/ClearTables/ClearTables.git
pushd ClearTables
make
createdb ct
psql -d ct -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore; CREATE EXTENSION unaccent; CREATE EXTENSION fuzzystrmatch;'
make # Create cleartables files
cat sql/types/*.sql | psql -1Xq -d ct # Load SQL types
osm2pgsql -E 3857 -G -d ct --number-processes 2 --output multi --style cleartables.json ~/path/to/extract # Load OSM data
cat sql/post/*.sql | psql -1Xq -d ct # Add post-import SQL
popd
```

Other osm2pgsql flags for performance or updates can be added, and will be necessary for large imports. See the osm2pgsql documentation for more details.
Flags that might be needed include
- `--slim`
- `--cache`
- `--flat-nodes`

Slim mode is not required by this style, so ``--slim --drop`` can be safely used if updates are not required.

If PostgreSQL [`max_connections`](http://www.postgresql.org/docs/9.3/static/runtime-config-connection.html#RUNTIME-CONFIG-CONNECTION-SETTINGS)
is increased from the default, `--number-processes` can be increased. If `--number-processes` is omitted, osm2pgsql will
attempt to use as many processes as hardware threads.

### Load coastline data

Meddo uses data from [OSMCoastline](https://github.com/osmcode/osmcoastline), hosted on [OpenStreetMapData](http://openstreetmapdata.com/), and from [Natural Earth](http://www.naturalearthdata.com/). The data used is documented in full in [`external-data.yml`](external-data.yml)

```sh
./get-external-data.py
```

It should be re-run periodically, and the command options can be obtained with `./get-external-data.py -h`

### Load admin data
Meddo uses data generated by [OSMBorder](https://github.com/pnorman/osmborder) for administrative boundaries.

Generate the borders with OSMBorder then run the SQL

```sql
CREATE TABLE osmborder_lines (
  osm_id bigint,
  admin_level int,
  disputed bool,
  maritime bool,
  way Geometry(LineString, 3857));

\copy osmborder_lines FROM osmborder_lines.csv

CREATE INDEX osmborder_lines_way_idx ON osmborder_lines USING gist (way) WITH (fillfactor=100);
CLUSTER osmborder_lines USING osmborder_lines_way_idx;
CREATE INDEX osmborder_lines_way_low_idx ON osmborder_lines USING gist (way) WITH (fillfactor=100) WHERE admin_level <= 4;
```

### Install required functions

Meddo requires some standard stylesheet-independent functions

```sh
psql -d ct -f functions.sql
```

## Usage

### Development

A suitable design program like Kosmtik or Mapbox Studio Classic is needed. If Kosmtik is installed, `kosmtik serve data.yml` will start Kosmtik, and the Data Inspector can be used. For Mapbox Studio, the entire repository is a tm2source project.

### Production

It might be necessary to compile the project to Mapnik XML for production, which can be done in many ways. One way is `kosmtik export data.yml --format xml --output meddo.xml`

## Schema

*TBD.*

## Why "meddo"?

[Meddo](https://en.wikipedia.org/wiki/Marshall_Islands_stick_chart#Meddo_charts) is a type of [Polynesian stick chart](https://en.wikipedia.org/wiki/Marshall_Islands_stick_chart) which contains only a section of the island chain, and meddo creates vector tiles from a larger database. It's also based on the same language as "wiki".

## License

The code is licensed under the [MIT License](LICENSE). If used as directed, use [ODbL licensed OpenStreetMap data](https://www.openstreetmap.org/copyright).

{'fraction_non_alphanumeric': 0.09289003706615748, 'fraction_numerical': 0.011681455689093564, 'mean_word_length': 4.540759178593653, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 16, 'lorem ipsum': 0, 'www.': 6, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4057782', 'n_tokens_mistral': 2807, 'n_tokens_neox': 2666, 'n_words': 1067}