<commit_before># Managing Datasets

Working with 3D (and 2D) image datasets is at the heart of webKnossos. 

- [Import datasets](#importing-datasets) by uploading them directly via the web UI or by using the file system (self-hosted instances only).
- [Configure the dataset](#configuring-datasets) defaults and permissions to your specification.
- [Share your datasets](./sharing.md#dataset-sharing) with the public or with selected users.

[Read the section on file and data formats](./data_formats.md) if you are interested in the technical background and concepts behind webKnossos datasets.

## Importing Datasets
webKnossos supports both loading data from a local hard disk or streaming it from a remote storage server or the cloud. In either case, webKnossos acts as the central hub to manage all your datasets in one place, search, and tag them. There are several ways to import and add a dataset to webKnossos:

### Uploading through the web browser
The easiest way to get started with working on your datasets is through the webKnossos web interface. You can directly upload your dataset through the browser.

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Provide some metadata information:
  - a *name* 
  - give access permissions for one or more teams (use the `default` team if all members of your organization should be able to see it)
  - *scale* of each voxel (in nanometers)
3. Drag and drop your data into the upload section
4. Click the *Upload* button


webKnossos uses the [WKW-format](./data_formats.md#wkw-datasets) internally to display your data.
If your data is already in WKW you can simply drag your folder (or zip archive of that folder) into the upload view.

If your data is not in WKW, you can either:

- upload the data in a supported file format and webKnossos will automatically convert it to WKW ([webknossos.org](https://webknossos.org) only). Depending on the size of the dataset, the conversion will take some time. You can check the progress at the "Jobs" page or the "Datasets" tab in the dashboard (both will update automatically).
- [Convert](#converting-datasets) your data manually to WKW.

In particular, the following file formats are supported for uploading (and conversion):

- [WKW dataset](#WKW-Datasets)
- [Image file sequence](#Single-Layer-Image-File-Sequence) in one folder (tif, jpg, png, dm3, dm4)
  - as an extension, multiple folders with image sequences are interpreted as [separate layers](#Multi-Layer-Image-File-Sequence)
- Single-file images (tif, czi, nifti, raw)
- KNOSSOS file hierarchy 
- [Read more about the supported file formats and details](./data_formats.md#conversion-with-webknossosorg)

Once the data is uploaded (and potentially converted), you can further configure a dataset's [Settings](#configuring-datasets) and double-check layer properties, finetune access rights & permissions, or set default values for rendering.

### Working with Zarr datasets
webKnossos supports loading and remotely streaming [Zarr](https://zarr.dev) datasets from a remote HTTP server or the cloud. webKnossos supports loading Zarr v2 datasets according to the [OME NGFF v0.4 spec](https://ngff.openmicroscopy.org/latest/).

webKnossos can load several Zarr sources and assemble them into a webKnossos dataset with several layers, e.g. one Zarr file/source for the `color` layer and one Zarr file/source for a `segmentation` layer.

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Select the *Add Remote Zarr Dataset*
3. For each layer, provide some metadata information:
  - a URL or domain/collection identifier to locate the dataset on the remote service
  - authentication credentials for accessing the resources on the remote service (optional)
4. Click the *Add Layer* button
5. webKnossos will automatically try to infer as many dataset properties (voxel size, bounding box, etc) as possible and preview a [webKnossos `datasource` configuration](./data_formats.md#dataset-metadata-specification) for your to review. 
  Consider setting the dataset `name` property and double-check all other properties for correctness.
6. Click `Import` to finish

webKnossos will NOT download/copy any data from these third-party data providers. 
Rather, any data viewed in webKnossos will be streamed read-only and directly from the remote source. 
Any other webKnossos feature, e.g., annotations, and access rights, will be stored in webKnossos and do not affect these services. 

Note that data streaming may count against any usage limits or minutes as defined by these third-party services. Check with the service provider or dataset owner.

Hint: If you happen to have any Zarr dataset locally that you would like to view in webKnossos, consider running an HTTP server locally to serve the dataset. Then webKnossos can easily stream the data.
Alternatively, convert the dataset to wkw using [webknossos-libs](https://github.com/scalableminds/webknossos-libs/).

### Working with Neuroglancer and BossDB datasets
webKnossos.org supports loading and remotely streaming datasets in the [Neuroglancer precomputed format](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed) stored in the Google Cloud or datasets served from [BossDB](https://bossdb.org).

To import these datasets:

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Select the *Add Neuroglancer Dataset* or *Add BossDB Dataset* tab
3. Provide some metadata information:
  - a *dataset name* 
  - a URL or domain/collection identifier to locate the dataset on the remote service
  - authentication credentials for accessing the resources on the remote service (optional)
4. Click the *Add* button

webKnossos will NOT download/copy any data from these third-party data providers. 
Rather, any data viewed in webKnossos will be streamed read-only and directly from the remote source. 
Any other webKnossos feature, e.g., annotations, and access rights, will be stored in webKnossos and do not affect these services. 

Note that data streaming may count against any usage limits or minutes as defined by these third-party services. Check with the service provider or dataset owner.

### Working with N5 datasets
We are working on integrating [N5](https://github.com/saalfeldlab/n5) support into webKnossos. If you have datasets in the N5 format and would like to work with us on building, testing, and refining the N5 integration into webKnossos then [please contact us](mailto:<EMAIL>).

### Uploading through the Python API
For those wishing to automate dataset upload or to do it programmatically, check out the webKnossos [Python library](https://github.com/scalableminds/webknossos-libs). It allows you to create, manage and upload datasets as well. 

### Uploading through the File System
-- (Self-Hosted Instances Only)-- 

On self-hosted instances, large datasets can be efficiently imported by placing them directly in the file system (WKW-format only):

* Place the dataset at `<webKnossos directory>/binaryData/<Organization name>/<Dataset name>`. For example `/opt/webknossos/binaryData/Springfield_University/great_dataset`.
* Go to the [dataset view on the dashboard](./dashboard.md)
* Use the refresh button on the dashboard or wait for webKnossos to detect the dataset (up to 10min)

Typically webKnossos can infer all the required metadata for a dataset automatically and import datasets automatically on refresh. In some cases, you will need to manually import a dataset and provide more information:
<commit_msg>markdown formatting fix for docs<commit_after># Managing Datasets

Working with 3D (and 2D) image datasets is at the heart of webKnossos. 

- [Import datasets](#importing-datasets) by uploading them directly via the web UI or by using the file system (self-hosted instances only).
- [Configure the dataset](#configuring-datasets) defaults and permissions to your specification.
- [Share your datasets](./sharing.md#dataset-sharing) with the public or with selected users.

[Read the section on file and data formats](./data_formats.md) if you are interested in the technical background and concepts behind webKnossos datasets.

## Importing Datasets
webKnossos supports both loading data from a local hard disk or streaming it from a remote storage server or the cloud. In either case, webKnossos acts as the central hub to manage all your datasets in one place, search, and tag them. There are several ways to import and add a dataset to webKnossos:

### Uploading through the web browser
The easiest way to get started with working on your datasets is through the webKnossos web interface. You can directly upload your dataset through the browser.

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Provide some metadata information:  
  - a *name* 
  - give access permissions for one or more teams (use the `default` team if all members of your organization should be able to see it)
  - *scale* of each voxel (in nanometers)
3. Drag and drop your data into the upload section
4. Click the *Upload* button


webKnossos uses the [WKW-format](./data_formats.md#wkw-datasets) internally to display your data.
If your data is already in WKW you can simply drag your folder (or zip archive of that folder) into the upload view.

If your data is not in WKW, you can either:

- upload the data in a supported file format and webKnossos will automatically convert it to WKW ([webknossos.org](https://webknossos.org) only). Depending on the size of the dataset, the conversion will take some time. You can check the progress at the "Jobs" page or the "Datasets" tab in the dashboard (both will update automatically).
- [Convert](#converting-datasets) your data manually to WKW.

In particular, the following file formats are supported for uploading (and conversion):

- [WKW dataset](#WKW-Datasets)
- [Image file sequence](#Single-Layer-Image-File-Sequence) in one folder (tif, jpg, png, dm3, dm4)
  - as an extension, multiple folders with image sequences are interpreted as [separate layers](#Multi-Layer-Image-File-Sequence)
- Single-file images (tif, czi, nifti, raw)
- KNOSSOS file hierarchy 
- [Read more about the supported file formats and details](./data_formats.md#conversion-with-webknossosorg)

Once the data is uploaded (and potentially converted), you can further configure a dataset's [Settings](#configuring-datasets) and double-check layer properties, finetune access rights & permissions, or set default values for rendering.

### Working with Zarr datasets
webKnossos supports loading and remotely streaming [Zarr](https://zarr.dev) datasets from a remote HTTP server or the cloud. webKnossos supports loading Zarr v2 datasets according to the [OME NGFF v0.4 spec](https://ngff.openmicroscopy.org/latest/).

webKnossos can load several Zarr sources and assemble them into a webKnossos dataset with several layers, e.g. one Zarr file/source for the `color` layer and one Zarr file/source for a `segmentation` layer.

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Select the *Add Remote Zarr Dataset*
3. For each layer, provide some metadata information:  
  - a URL or domain/collection identifier to locate the dataset on the remote service
  - authentication credentials for accessing the resources on the remote service (optional)
4. Click the *Add Layer* button
5. webKnossos will automatically try to infer as many dataset properties (voxel size, bounding box, etc) as possible and preview a [webKnossos `datasource` configuration](./data_formats.md#dataset-metadata-specification) for your to review. 
  Consider setting the dataset `name` property and double-check all other properties for correctness.
6. Click `Import` to finish

webKnossos will NOT download/copy any data from these third-party data providers. 
Rather, any data viewed in webKnossos will be streamed read-only and directly from the remote source. 
Any other webKnossos feature, e.g., annotations, and access rights, will be stored in webKnossos and do not affect these services. 

Note that data streaming may count against any usage limits or minutes as defined by these third-party services. Check with the service provider or dataset owner.

Hint: If you happen to have any Zarr dataset locally that you would like to view in webKnossos, consider running an HTTP server locally to serve the dataset. Then webKnossos can easily stream the data.
Alternatively, convert the dataset to wkw using [webknossos-libs](https://github.com/scalableminds/webknossos-libs/).

### Working with Neuroglancer and BossDB datasets
webKnossos.org supports loading and remotely streaming datasets in the [Neuroglancer precomputed format](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed) stored in the Google Cloud or datasets served from [BossDB](https://bossdb.org).

To import these datasets:

1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
2. Select the *Add Neuroglancer Dataset* or *Add BossDB Dataset* tab
3. Provide some metadata information:  
  - a *dataset name* 
  - a URL or domain/collection identifier to locate the dataset on the remote service
  - authentication credentials for accessing the resources on the remote service (optional)
4. Click the *Add* button

webKnossos will NOT download/copy any data from these third-party data providers. 
Rather, any data viewed in webKnossos will be streamed read-only and directly from the remote source. 
Any other webKnossos feature, e.g., annotations, and access rights, will be stored in webKnossos and do not affect these services. 

Note that data streaming may count against any usage limits or minutes as defined by these third-party services. Check with the service provider or dataset owner.

### Working with N5 datasets
We are working on integrating [N5](https://github.com/saalfeldlab/n5) support into webKnossos. If you have datasets in the N5 format and would like to work with us on building, testing, and refining the N5 integration into webKnossos then [please contact us](mailto:<EMAIL>).

### Uploading through the Python API
For those wishing to automate dataset upload or to do it programmatically, check out the webKnossos [Python library](https://github.com/scalableminds/webknossos-libs). It allows you to create, manage and upload datasets as well. 

### Uploading through the File System
-- (Self-Hosted Instances Only)-- 

On self-hosted instances, large datasets can be efficiently imported by placing them directly in the file system (WKW-format only):

* Place the dataset at `<webKnossos directory>/binaryData/<Organization name>/<Dataset name>`. For example `/opt/webknossos/binaryData/Springfield_University/great_dataset`.
* Go to the [dataset view on the dashboard](./dashboard.md)
* Use the refresh button on the dashboard or wait for webKnossos to detect the dataset (up to 10min)

Typically webKnossos can infer all the required metadata for a dataset automatically and import datasets automatically on refresh. In some cases, you will need to manually import a dataset and provide more information:

{'fraction_non_alphanumeric': 0.06333355232901912, 'fraction_numerical': 0.0036791275211878324, 'mean_word_length': 4.686215913335824, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2174555', 'n_tokens_mistral': 4135, 'n_tokens_neox': 3913, 'n_words': 2080}