<commit_before># raster-object-recognition

## What is it?

Object recognition within rasters using machine learning. This is adapted
from aresearch project that evaluated the how to identify buildings in NAIP
DOQQ imagery. It uses various OpenSource tools and Python scripts to be a
work flow for this task. The whole process can be run from a Docker
container.

## The Work Flow

This is the basic work flow at a high level.

1. Download NAIP imagery for area of interest
2. Download some Census Tiger data and other misc GIS data
3. Build a training dataset of buildings
4. Segment the imagery for a training area
5. Build a training dataset based on 40% of training area
6. Run search in the 40% of training area
7. Review results of search
8. If results are not satisfactory, adjust parameters and goto 4.
9. Do a final check on remaining 20% of training area, Review and Loop is needed
10. Segment target search area
11. Run search in target search area
12. Review results
13. Post process the results

## Requirements

The processing pipeline system is built on Ubuntu 16.04 in a Docker container.
The Docker container is not required if you are already running Ubuntu 16.04+
or want to resolve the dependencies on your host system. See the file
*Dockerfile* for the list of packages to install.

### Use of host services

* Disk space through docker volume maps
* PostgreSQL database - The system makes extensive use of PostgreSQL database
* A Webserver - (optional) The current system as tools for visualizing data using mapserver and OpenLayers.

The database and mapserver could be configured to run inside the docker
container. There might be more information on this in the docs directory.

Lots of disk space. This depends on your area of interest, but the space is
needed for the NAIP imagery, segmentation polygons, database tables, etc.
The NAIP imagery for all of California was about 4TB for the raw data, We
compressed that down to about 1TB for a working set. In addition to that
we had about 1TB in PostgreSQL tables. Regardless, plan for this and monitor
your usage.

## Directory Structure
<commit_msg>Adding info on docker runtime requirements.<commit_after># raster-object-recognition

## What is it?

Object recognition within rasters using machine learning. This is adapted
from aresearch project that evaluated the how to identify buildings in NAIP
DOQQ imagery. It uses various OpenSource tools and Python scripts to be a
work flow for this task. The whole process can be run from a Docker
container.

## The Work Flow

This is the basic work flow at a high level.

1. Download NAIP imagery for area of interest
2. Download some Census Tiger data and other misc GIS data
3. Build a training dataset of buildings
4. Segment the imagery for a training area
5. Build a training dataset based on 40% of training area
6. Run search in the 40% of training area
7. Review results of search
8. If results are not satisfactory, adjust parameters and goto 4.
9. Do a final check on remaining 20% of training area, Review and Loop is needed
10. Segment target search area
11. Run search in target search area
12. Review results
13. Post process the results

## Requirements

The processing pipeline system is built on Ubuntu 16.04 in a Docker container.
The Docker container is not required if you are already running Ubuntu 16.04+
or want to resolve the dependencies on your host system. See the file
*Dockerfile* for the list of packages to install.

### Docker

If you don't have Docker installed on your system, then go
[HERE](https://www.docker.com/) and find the downloads or instsrtuctions for
your system and install the Docker runtime.

There example scripts to build a Docker image and to run the Docker image in
``src/docker/``.

### Use of host services

* Disk space through docker volume maps
* PostgreSQL database - The system makes extensive use of PostgreSQL database
* A Webserver - (optional) The current system as tools for visualizing data using mapserver and OpenLayers.

The database and mapserver could be configured to run inside the docker
container. There might be more information on this in the docs directory.

Lots of disk space. This depends on your area of interest, but the space is
needed for the NAIP imagery, segmentation polygons, database tables, etc.
The NAIP imagery for all of California was about 4TB for the raw data, We
compressed that down to about 1TB for a working set. In addition to that
we had about 1TB in PostgreSQL tables. Regardless, plan for this and monitor
your usage.

## Directory Structure

{'fraction_non_alphanumeric': 0.036323851203501095, 'fraction_numerical': 0.015317286652078774, 'mean_word_length': 4.176670441676104, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '271970', 'n_tokens_mistral': 1223, 'n_tokens_neox': 1104, 'n_words': 752}