<jupyter_start><jupyter_text>Kili Tutorial: Importing medical data into a frame project  In this tutorial, we will show you how to import dicom data into a [Frame Kili project](https://cloud.kili-technology.com/docs/video-interfaces/multi-frames-classification/docsNav). Such projects allow you to annotate volumes of image data.The data we use comes from [The Cancer Genome Atlas Lung Adenocarcinoma (TCGA-LUAD) data collection](https://wiki.cancerimagingarchive.net/display/Public/TCGA-LUAD). We selected 3 scans out of this dataset.  Downloading data Let's first import the scans. We host these files in a .zip on GDrive.<jupyter_code>import os
import subprocess

import tqdm
os.makedirs(os.path.expanduser('~/Downloads'), exist_ok=True)<jupyter_output><empty_output><jupyter_text>We will use a small package to help downloading the file hosted on Google Drive<jupyter_code>%%bash
git clone https://github.com/chentinghao/download_google_drive.git
cd download_google_drive/
python download_gdrive.py 1q3qswXthFh3xMtAAnePph6vav3N7UtOF ~/Downloads/dicom_files.zip
!apt-get install unzip
!unzip ~/Downloads/dicom_files.zip -d ~/Downloads/ > /dev/null<jupyter_output><empty_output><jupyter_text>Reading data We can then read the dicom files with [pydicom](https://pydicom.github.io/pydicom/stable/).<jupyter_code>ASSET_ROOT = os.path.expanduser('~/Downloads/TCGA-LUAD')

sorted_files = {}
asset_number = 0
for root, dirs, files in os.walk(ASSET_ROOT):
    if len(files) > 0:
        file_paths = list(map(lambda path: os.path.join(root, path), files))
        sorted_files[f'asset-{asset_number+1}'] = sorted(file_paths,
            key=lambda path: int(path.split('/')[-1].split('-')[1].split('.')[0]))
        asset_number += 1<jupyter_output><empty_output><jupyter_text>Let's see what is inside the dataset :<jupyter_code>!pip install pillow
!pip install pydicom
from PIL import Image
import pydicom

def read_dcm_image(path):
    dicom = pydicom.dcmread(path)
    image = dicom.pixel_array
    # Currently, Kili does not support windowing in the application.
    # This will soon change, but until then we advise you to reduce the range to 256 values.
    image = (image - image.min()) / (image.max() - image.min()) * 256
    return Image.fromarray(image).convert('RGB')

for asset_key in sorted_files.keys():
    print(asset_key)
    im = read_dcm_image(sorted_files[asset_key][20])
    display(im)<jupyter_output>Requirement already satisfied: pillow in /usr/local/Caskroom/miniconda/base/envs/kilienv/lib/python3.7/site-packages (8.1.1)
Requirement already satisfied: pydicom in /usr/local/Caskroom/miniconda/base/envs/kilienv/lib/python3.7/site-packages (2.1.2)
asset-1<jupyter_text>Extracting and serving images For each of the dicom `.dcm` files, let's extract its content (image) and save it into a `.jpeg` image.<jupyter_code>sorted_images = {}
for asset_key, files in sorted_files.items():
    images = []
    for file in tqdm.tqdm(files):
        im = read_dcm_image(file)
        im_file = file.replace('.dcm', '.jpeg')
        im.save(im_file, format='JPEG')
        images.append(im_file)
    sorted_images[asset_key] = images<jupyter_output>100%|██████████| 201/201 [00:02<00:00, 78.70it/s]
100%|██████████| 227/227 [00:02<00:00, 88.36it/s]
100%|██████████| 329/329 [00:04<00:00, 77.39it/s]<jupyter_text>We now have extracted jpeg images processable by Kili.  Creating the project We can now import those assets into a FRAME project !Let's begin by creating a project<jupyter_code>## You can also directly create the interface on the application.
interface = {
	"jobRendererWidth": 0.17,
	"jobs": {
		"JOB_0": {
			"mlTask": "OBJECT_DETECTION",
			"tools": [
				"semantic"
			],
			"instruction": "Segment the right class",
			"required": 1,
			"isChild": False,
			"content": {
				"categories": {
					"BONE": {
						"name": "Bone",
						"children": [],
						"color": "#0755FF"
					},
					"LUNG": {
						"name": "Lung",
						"children": [],
						"color": "#EEBA00"
					},
					"TISSUE_0": {
						"name": "Tissue",
						"children": [],
						"color": "#941100"
					}
				},
				"input": "radio"
			}
		}
	}
}
## Authentication
from kili.client import Kili

api_key = os.getenv('KILI_USER_API_KEY')
api_endpoint = os.getenv('KILI_API_ENDPOINT') # If you use Kili SaaS, use the url 'https://cloud.kili-technology.com/api/label/v2/graphql'
kili = Kili(api_key=api_key, api_endpoint=api_endpoint)

## Project creation
project = kili.create_project(
    description='Demo FRAME project',
    input_type='FRAME',
    json_interface=interface,
    title='Lungs from TCGA-LUAD'
)
project_id = project['id']<jupyter_output><empty_output><jupyter_text>Importing images Finally, let's import the volumes using `appendManyToDataset` (see [link](https://staging.cloud.kili-technology.com/docs/python-graphql-api/python-api/append_many_to_dataset)). The key argument is `json_content_array`, which is a list of list of strings. Each element is the list of urls or paths pointing to images of the volume considered. - Let's host these images locally to demonstrate how we would do it with cloud URLs for example :<jupyter_code>subprocess.Popen(f'python -m http.server 8001 --directory {ASSET_ROOT}',
                 shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)
ROOT_URL = 'http://localhost:8001/'
def files_to_urls(files):
    return list(map(lambda file: ROOT_URL + file.split('TCGA-LUAD')[1], files))
kili.append_many_to_dataset(
    project_id=project_id,
    external_id_array=list(sorted_images.keys()),
    json_content_array=list(map(files_to_urls, sorted_images.values()))
)<jupyter_output><empty_output><jupyter_text>Or, as mentionned, you can simply provide the paths to your images, and call the function like below :<jupyter_code>kili.append_many_to_dataset(
    project_id=project_id,
    external_id_array=list(map(lambda key: f'local-path-{key}',sorted_images.keys())),
    json_content_array=list(sorted_images.values())
)<jupyter_output><empty_output><jupyter_text>Back to the interface We can see our assets were imported...<jupyter_code>ds_size = kili.count_assets(project_id=project_id)
print(ds_size)
assert ds_size == 6<jupyter_output>6
{'fraction_non_alphanumeric': 0.12976113621691412, 'fraction_numerical': 0.020335700451904453, 'mean_word_length': 4.680109990834097, 'pattern_counts': {'":': 23, '<': 37, '<?xml version=': 0, '>': 36, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '656777', 'license': 'Apache-2.0', 'n_tokens_mistral': 2369, 'n_tokens_neox': 2141, 'n_words': 515, 'path': 'recipes/frame_dicom_data.ipynb', 'repo_name': 'kili-technology/kili-playground', 'size': '6196'}