<commit_before># Data Load and Access

## Load

Piazza provides the ability to load external data into the system: metadata is extracted from external data, stored within Piazza, and a resource ID is returned. The metadata is also entered into Piazza's search index. Piazza supports several data formats today -- including GeoJSON, Shapefies, and GeoTIFFs -- with more to come as users require them.

For example, the URL of a GeoTIFF stored in an S3 bucket can be sent to Piazza and, once loaded, Piazza can perform other operations on the data such as generating a WMS layer or sending the data to a user service. The metadata for the file will include the S3 URL, and so the resource ID can be used as a global, unique reference to the data.

Piazza is not intended to be a storage system for user data and so normally only the metadata is stored -- not the file itself. We refer to this as the _no-host_ model.

In the no-host case, Piazza will need to have read-access to the file and in some cases will have to copy the file to temporary local storage in order to open the file and extract the metadata; when the extraction is complete, the file is deleted. For large files, this will incur a performance penalty.

Although not intended for long-term use, Piazza does also support a _hosted_ model in which the data is copied locally for metadata extraction but _not_ deleted.

## Hosted Image File Example

TODO: explain the JSON input and output fields

This example shows how to load a GeoTIFF file from your local file system into Piazza, with `host` set to `true`.
The script will print out the id of the _job_ that was submitted.

NOTE: The GeoTIFF file used in these examples can be found at link:scripts/terrametrics.tif[terrametrics.tif]. Run this script using `sh 4-file-load.sh` with terrametrics.tif in your present working directory.

[source,sh]
.link:scripts/4-file-load.sh[4-file-load.sh]
----
include::scripts/4-file-load.sh[tags=public]
----

*TODO: explain JSON input and output fields*

A client can now periodically check the status of the job, waiting for it to be completed. This script takes as a single command-line parameter, the job id string. Run this script using `sh 4-file-job.sh {jobId}` (jobId is displayed by the previous script).

[source,sh]
.link:scripts/4-file-job.sh[4-file-job.sh]
----
include::scripts/4-file-job.sh[tags=public]
----


Once the job is complete, we can retrieve the metadata about the image. Here too the script takes an argument: this time,
the id of the data resource (displayed by the previous script).

[source,sh]
.link:scripts/4-file-info.sh[4-file-info.sh]
----
include::scripts/4-file-info.sh[tags=public]
----

Next, we can retrieve the file:

[source,sh]
.link:scripts/4-file-download.sh[4-file-download.sh]
----
include::scripts/4-file-download.sh[tags=public]
----

And, finally, we can create a WMS endpoint for our GeoTIFF:

[source,sh]
.link:scripts/4-file-wms.sh[4-file-wms.sh]
----
include::scripts/4-file-wms.sh[tags=public]
----


## Non-hosted Image File

TODO: load, get data id, download, wms


## Hosted GeoJSON File

TODO: load, get data id, metadata, download, wfs<commit_msg>Update 4-load-access.txt<commit_after># Data Load and Access

## Load

Piazza provides the ability to load external data into the system: metadata is extracted from external data, stored within Piazza, and a resource ID is returned. The metadata is also entered into Piazza's search index. Piazza supports several data formats today -- including GeoJSON, Shapefies, and GeoTIFFs -- with more to come as users require them.

For example, the URL of a GeoTIFF stored in an S3 bucket can be sent to Piazza and, once loaded, Piazza can perform other operations on the data such as generating a WMS layer or sending the data to a user service. The metadata for the file will include the S3 URL, and so the resource ID can be used as a global, unique reference to the data.

Piazza is not intended to be a storage system for user data and so normally only the metadata is stored -- not the file itself. We refer to this as the _no-host_ model.

In the no-host case, Piazza will need to have read-access to the file and in some cases will have to copy the file to temporary local storage in order to open the file and extract the metadata; when the extraction is complete, the file is deleted. For large files, this will incur a performance penalty.

Although not intended for long-term use, Piazza does also support a _hosted_ model in which the data is copied locally for metadata extraction but _not_ deleted.


## Hosted GeoTIFF File Example

TODO: explain the JSON input and output fields

This example shows how to load a GeoTIFF file from your local file system into Piazza, with `host` set to `true`.
The script will print out the id of the _job_ that was submitted.

NOTE: The GeoTIFF file used in these examples can be found at link:scripts/terrametrics.tif[terrametrics.tif]. Run this script using `sh 4-hosted-load.sh` with terrametrics.tif in your present working directory.

[source,sh]
.link:scripts/4-hosted-load.sh[4-hosted-load.sh]
----
include::scripts/4-hosted-load.sh[tags=public]
----

*TODO: explain JSON input and output fields*

A client can now periodically check the status of the job, waiting for it to be completed. This script takes as a single command-line parameter, the job id string. Run this script using `sh 4-job.sh {jobId}` (`{jobId}` is displayed by the previous script).

[source,sh]
.link:scripts/4-job.sh[4-job.sh]
----
include::scripts/4-job.sh[tags=public]
----

Finally, we can retrieve the file, using the id of the data resource (displayed by the previous script): `sh 4-hosted-download.sh {dataId}`.

[source,sh]
.link:scripts/4-hosted-download.sh[4-hosted-download.sh]
----
include::scripts/4-hosted-download.sh[tags=public]
----


## Non-hosted GeoJSON File

This example shows how to load an externally hosted file into piazza, without piazza hosting the file itself.
The script will print out the id of the _job_ that was submitted.

[source,sh]
.link:scripts/4-nonhosted-load.sh[4-nonhosted-load.sh]
----
include::scripts/4-nonhosted-load.sh[tags=public]
----

After this, we can get the dataId using the jobId in the same way we do in the hosted version:

.link:scripts/4-job.sh[4-job.sh].
----
include::scripts/4-job.sh[tags=public]
----

Using the dataId (displayed by the previous script), we can create a WMS endpoint for the file.

.link:scripts/4-nonhosted-wms.sh[4-nonhosted-wms.sh].
----
include::scripts/4-nonhosted-wms.sh[tags=public]
----
{'fraction_non_alphanumeric': 0.0850609756097561, 'fraction_numerical': 0.0065548780487804876, 'mean_word_length': 4.000762195121951, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6422729', 'n_tokens_mistral': 1962, 'n_tokens_neox': 1892, 'n_words': 920}