<jupyter_start><jupyter_text>Profiling with Different Studies (DINED Profiler functionality)In this notebook, we will try to find out where a certain person is positioned within a study. We will do this for two studies, so that we also demonstrate how to convert different data structures to the common DINED format.The dimensions are derived from two dataset of human body measurements:* DINED delstu2016 - https://doi.org/10.4121/uuid:1411a8ac-5944-41d2-81c1-1edf32106d99* DINED adults2004 - https://data.4tu.nl/ndownloader/files/24015182 DatasetWe first retrieve the dataset from the 4TU data repository:<jupyter_code>!wget -nc -O data.zip https://data.4tu.nl/ndownloader/files/23994179
!unzip -o data.zip delstu2016.csv
!rm data.zip

!wget -nc -O data.zip https://data.4tu.nl/ndownloader/files/24015182
!unzip -o data.zip dined2004.csv
!rm data.zip<jupyter_output>--2022-04-07 14:47:14--  https://data.4tu.nl/ndownloader/files/23994179
Resolving data.4tu.nl (data.4tu.nl)... 172.16.17.32
Connecting to data.4tu.nl (data.4tu.nl)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘data.zip’

data.zip                [ <=>                ]  26,38K  --.-KB/s    in 0,02s   

2022-04-07 14:47:14 (1,60 MB/s) - ‘data.zip’ saved [27010]

Archive:  data.zip
  inflating: delstu2016.csv          
--2022-04-07 14:47:15--  https://data.4tu.nl/ndownloader/files/24015182
Resolving data.4tu.nl (data.4tu.nl)... 172.16.17.32
Connecting to data.4tu.nl (data.4tu.nl)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘data.zip’

data.zip                [ <=>                ]  56,38K  --.-KB/s    in 0,04s   

2022-04-07 14:47:15 (1,31 MB/s) - ‘data.zip’ saved [57738]

Archive:  data.zip
  inflating: dined2004.csv[...]<jupyter_text>Now let's load the measurement descriptions from DINED:<jupyter_code>import numpy as np
import pandas as pd
import json

class Measure():
    def __init__(self, id: int, name: str, description: str, unit: str):
        self.id = id
        self.name = name
        self.description = description
        self.unit = unit

    def __str__(self):
        return "Measure: " + str(self.id) + " " + self.name + " (" + self.unit + ")"


class MeasureGroup():
    def __init__(self, id: int, name: str, measures: list[Measure]):
        self.id = id
        self.name = name
        self.measures = measures

    def __str__(self):
        lines = ["Measure Group: " + str(self.id) + " " + self.name]
        for m in self.measures:
            lines += [ "\t"+str(m)]
        return "\n\t".join(lines)


class MeasureCollection():
    def __init__(self, name: str, description: str, measure_groups: list[MeasureGroup]):
        self.name = name
        self.description = description
        self.measure_groups = measure_groups
    
    def __str__(self):
        lines = ["Measure Collection: " + self.name]
        for m in self.measure_groups:
            lines += [  str(m)]
        return "\n\t".join(lines)

def json_decoder(d):
    id = int(d.get("id"))
    name = d.get("name_en")
    unit = d.get("dimension")
    description = d.get("description_en")
    labels = d.get("labels")

    if unit:
        return Measure(id,name,description,unit)
    elif labels:
        sorted_labels = sorted(labels, key=lambda x: x.id)
        return MeasureGroup(id,name,sorted_labels)
    else:
        return None

with open("assets/measurements.json") as json_file:
    measure_collection = MeasureCollection("DINED Measures","Measures from the DINED website",None)
    groups = json.load(json_file, object_hook=json_decoder)
    measure_collection.measure_groups = sorted(groups, key=lambda x: x.id)<jupyter_output><empty_output><jupyter_text>Let's pretty print the set of DINED measures:<jupyter_code>print(measure_collection)<jupyter_output>Measure Collection: DINED Measures
	Measure Group: 1 Standing, length/depth
		Measure: 1 Reach height, standing (mm)
		Measure: 2 Stature (mm)
		Measure: 3 Eye height, standing (mm)
		Measure: 4 Shoulder height (mm)
		Measure: 5 Elbow height, standing (mm)
		Measure: 6 Fist height, standing (mm)
		Measure: 8 Hip height (mm)
		Measure: 19 Reach depth (mm)
		Measure: 20 Arm length (mm)
		Measure: 27 Chest depth (mm)
		Measure: 79 Palm height, standing (mm)
		Measure: 81 Crotch height (mm)
	Measure Group: 2 Standing, width/circumference
		Measure: 21 Breadth over the elbows (mm)
		Measure: 22 Shoulder breadth (bi-deltoid) (mm)
		Measure: 24 Hip breadth (mm)
		Measure: 26 Knee breadth (mm)
		Measure: 37 Chest circumference (mm)
		Measure: 38 Waist circumference (mm)
		Measure: 39 Hip circumference (mm)
		Measure: 40 Upperarm circumference (mm)
		Measure: 82 Calf circumference (mm)
		Measure: 83 Thigh circumference (mm)
	Measure Group: 3 Sitting, length/width/depth
		Measure: 9 Thigh cleara[...]
{'fraction_non_alphanumeric': 0.1219264377159114, 'fraction_numerical': 0.05730542572647836, 'mean_word_length': 3.2688638334778837, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '426343', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1886, 'n_tokens_neox': 1622, 'n_words': 472, 'path': 'notebooks/ProfilingDifferentStudies.ipynb', 'repo_name': 'dined-io/dyned', 'size': '4929'}