<jupyter_start><jupyter_text>Auto Generate Lab measurement Dataset---------------------------A typical data source found in most industrial work places are lab measurements. This type of data usually has a well defined structure and is a mixture of categorical and numeric. Scale of measurements and the file format can vary greatly. This article covers how to generate a simple dataset that can be customized to simulate a wide range of use cases.    Contents1. data structure1. Packages used    - Faker    - xml.etree.ElementTree    - Pandas2. Generate dataset3. Convert to Pandas DataFrame4. Save File    - csv    - xml    - parquet  Sample Data Structure | Field          | DataType    | Min  | Max  ||----------------|-------------|------|------|| machine_id     | string      |      |      || test_id        | uuid        |      |      || technician     | string      |      |      || test_routine   | categorical |      |      || batched        | categorical |      |      || loc_1          | dictionary  |      |      || loc_1_x_offset | decimal     | -15  | 15   || loc_1_y_offset | decimal     | -1   | 1    || loc_1_z_offset | int         | 2500 | 5000 || loc_2          | dictionary  |      |      || loc_2_x_offset | decimal     | -15  | 15   || loc_2_y_offset | decimal     | -1   | 1    || loc_2_z_offset | int         | 2500 | 5000 |  Packages Used**Faker:** Is a python package for creating, as the name implies, fake data. This libriray not only generates random numbers and strings but more complex elements like addresses, and names.<jupyter_code>from faker import Faker
fake = Faker()
Faker.seed(0)<jupyter_output><empty_output><jupyter_text>**xml.etree.ElementTree:** Is a base library of Python that is a simple and efficient way of querying, parsing, and creating XML data.<jupyter_code>import xml.etree.ElementTree as ET
from xml.dom import minidom<jupyter_output><empty_output><jupyter_text>**Pandas:** is the goto library for working with table-like data.<jupyter_code>import pandas as pd
# base imports
import uuid
import os<jupyter_output><empty_output><jupyter_text>Generate DatasetThe structure of the dataset is reprecented by a dictionary and uses Faker to fill in the data values. This allows for simple modifications when moving between use cases.<jupyter_code>def generate_measurement_record():
    base_equipment_name='Machine'
    equipment_cnt = 25
    n = fake.pyint(min_value=1, max_value=equipment_cnt)

    measurement_record = {
        'machine_id': '_'.join([base_equipment_name, f'{n:02}'])
        ,'test_id':str(uuid.uuid4().hex)
        ,'technician':fake.name()
        ,'test_routine':fake.random_sample(elements=('a', 'b', 'c', 'd', 'e', 'f'))
        ,'batched':fake.random_sample(elements=('Yes', 'No', 'N/A'), length=1)[0]
        ,'loc_1':{
            'x_offset':fake.pydecimal(left_digits=2, right_digits=2, min_value=-15, max_value=15)
            ,'y_offset':fake.pydecimal(left_digits=1, right_digits=6, min_value=-1, max_value=1)
            ,'z_offset':fake.pyint(min_value=2500, max_value=5000)
        }
        ,'loc_2':{
            'x_offset':fake.pydecimal(left_digits=2, right_digits=2, min_value=-15, max_value=15)
            ,'y_offset':fake.pydecimal(left_digits=1, right_digits=6, min_value=-1, max_value=1)
            ,'z_offset':fake.pyint(min_value=2500, max_value=5000)
        }
    }
    return measurement_record<jupyter_output><empty_output><jupyter_text>Repetedly calling this function generates new records with random data that can be collected in a list up to a desired size.<jupyter_code>records_cnt = 5

measurement_record = None
measurement_list=[]

for _ in range(records_cnt):
    measurement_record = generate_measurement_record()
    measurement_list.append(measurement_record) # List of nested dictionarie<jupyter_output><empty_output><jupyter_text>Convert Dataset to XMLNext we'll need a function that will handle converting the records stored as dictionaries to xml. Then apply the function to each record in the list.<jupyter_code>import xml.etree.ElementTree as ET

def dict_to_xml(d, r=None):
    file_id = str(uuid.uuid4().hex)
    if r is None:
        r = ET.Element('DataFile')
        r.set('id', file_id)
    if isinstance(d, dict):
        for k, v in d.items():
            s = ET.SubElement(r, k)
            dict_to_xml(v, s)
    elif isinstance(d, tuple) or isinstance(d, list):
        val = '/'.join(str(v) for v in d)
        r.text = val
    elif isinstance(d, str):
        r.text = d
    else:
        r.text = str(d)
    return r
# Convert each record
xml_dataset = ET.Element('DataFiles')
for rcd in measurement_list:
    file_id = str(uuid.uuid4().hex)
    child = ET.SubElement(xml_dataset,'DataFile')
    child.set('id', file_id)
    
    # Convert to xml
    element= dict_to_xml(rcd,child)<jupyter_output><empty_output><jupyter_text>The resulting xml file can be viewed with:<jupyter_code>from xml.dom import minidom

xml= ET.tostring(xml_dataset, encoding='unicode', method='xml')
xml= minidom.parseString(xml)
xml= xml.toprettyxml(indent='  ')

print(xml)<jupyter_output><?xml version="1.0" ?>
<DataFiles>
  <DataFile id="33c768494e484df9b954c4c7bae2ce68">
    <machine_id>Machine_10</machine_id>
    <test_id>3c0c95f773bf426585a3a68642d2d41a</test_id>
    <technician><NAME></technician>
    <test_routine>c/f/a/b</test_routine>
    <batched>Yes</batched>
    <loc_1>
      <x_offset>8.63</x_offset>
      <y_offset>0.39661</y_offset>
      <z_offset>3736</z_offset>
    </loc_1>
    <loc_2>
      <x_offset>-4.94</x_offset>
      <y_offset>0.964363</y_offset>
      <z_offset>3182</z_offset>
    </loc_2>
  </DataFile>
</DataFiles><jupyter_text>Convert Dataset to Pandas DataFramePandas can directly create a DataFrame from a dictionary if the dictionary is not nested. So, the first step the convertion is to create a function to flatten the measurement dictionary structure:<jupyter_code>import collections.abc

def flatten(d, parent_key='', sep='_'):
    items = []
    for k, v in d.items():
        new_key = parent_key + sep + k if parent_key else k
        if isinstance(v, collections.MutableMapping):
            items.extend(flatten(v, new_key, sep=sep).items())
        else:
            items.append((new_key, v))
    return dict(items)<jupyter_output><empty_output><jupyter_text>Create a new list of flattened data:<jupyter_code>measurement_flat_list= []

for rcd in measurement_list:
    measurement_flat_list.append(flatten(rcd)) # List of single depth dictionaries<jupyter_output><empty_output><jupyter_text>Convert to DataFrame:<jupyter_code>measurement_df = pd.DataFrame(measurement_flat_list)
measurement_df.head()<jupyter_output><empty_output><jupyter_text>Save Dataset Files- XML- CSV- Parquet<jupyter_code># xml
destPath = 'sample_lab_measurement.xml'
# Convert to string
xml_str= ET.tostring(xml_dataset, encoding='unicode', method='xml')

# Pretty print string 
xml_str= minidom.parseString(xml_str)
xml_str= xml_str.toprettyxml(indent='  ')

# Write file
with open(destPath, 'w') as f:  # Write in file as utf-8
    f.write(xml_str)
# csv
destPath = 'sample_lab_measurement.csv'
measurement_df.to_csv(destPath, index = False, header=True)
# parquet
destPath = 'sample_lab_measurement.parquet'
measurement_df.to_parquet(destPath, index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11817685589519651, 'fraction_numerical': 0.02306222707423581, 'mean_word_length': 3.6268939393939394, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 1, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 36}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '171803', 'license': 'MIT', 'n_tokens_mistral': 2527, 'n_tokens_neox': 2409, 'n_words': 597, 'path': 'assets/2021-03-24/measurement-dataset-generator.ipynb', 'repo_name': 'DarrylBrysonDev0/cookbook', 'size': '7332'}