<jupyter_start><jupyter_text>This dataset is created using the template for creating a dataset from scratch as in: https://cookiecutter-easydata.readthedocs.io/en/latest/New-Dataset-Template/.<jupyter_code># Basic utility functions
import logging
import os
import pathlib
from pprint import pprint

from src.log import logger
from src import paths
from src.utils import list_dir
from functools import partial

# data functions
from src.data import DataSource, Dataset, DatasetGraph, Catalog
from src import helpers
# Optionally set to debug log level
logger.setLevel(logging.DEBUG)<jupyter_output><empty_output><jupyter_text>Create the Datasource<jupyter_code>ds_name = 'dataset-challenge'
dsrc = DataSource(ds_name)
license = """
CC-BY-4.0 is a common form of dataset license. Here you would put the license for your data, along with any attribution and other information necessary to keep in line with the terms included in the original license.

All data that you use should have an explicit license kept with it. To keep the license with the data, Easydata uses a license as one of its metadata fields which can be accessed via

`.LICENSE`

for any Dataset object.

For more on licenses, see the references at the end of the `04-Data-Challenge` notebook.
"""
descr = """
The `.DESCR` is where Easydata keeps a description of the dataset. In this example, you'll see that we have a Dataset object container with metadata, but no data.

For this dataset, if you do a ds.data, you will return NONE. 

A basic description of the data is something that always stays with the data, Easydata uses a descr as one of its metadata fields which can be accessed via

`.DESCR`

for any Dataset object.

When you transform the data, it is nice to append new information including what has been done to the data via the transformation by appending information to the end of the `.DESCR` text.

You can add any metadata you want to ds.metdata, as it is basically a dict with a fancy wrapping paper that lets you access any key via ALL CAPS.
"""
dsrc.add_metadata(contents=descr, force=True)
dsrc.add_metadata(contents=license, kind='LICENSE', force=True)
from src.data.extra import process_extra_files
process_function = process_extra_files
process_function_kwargs = {'file_glob':'*.csv',
                           'do_copy': True,
                           'extra_dir': ds_name+'.extra',
                           'extract_dir': ds_name}


dsrc.process_function = partial(process_function, **process_function_kwargs)
dsrc.update_catalog()<jupyter_output>2021-10-26 10:12:51,807 - catalog - DEBUG - Loaded 2 records from 'datasources' Catalog.
2021-10-26 10:12:51,810 - catalog - DEBUG - Verifying serialization for catalog 'datasources'
2021-10-26 10:12:51,815 - catalog - DEBUG - Writing entry:'dataset-challenge' to catalog:'datasources'.
2021-10-26 10:12:51,817 - datasets - DEBUG - Updated datasource:dataset-challenge in catalog<jupyter_text>Create the Corresponding Dataset<jupyter_code>from src.data import DatasetGraph
dag = DatasetGraph(catalog_path=paths['catalog_path'])
dag.add_source(output_dataset=ds_name, datasource_name=ds_name, overwrite_catalog=True)
ds = Dataset.from_catalog(ds_name)
ds = Dataset.load(ds_name)
print(ds.DESCR)
print(ds.LICENSE)
ds.EXTRA
ds.data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07566585956416465, 'fraction_numerical': 0.02209443099273608, 'mean_word_length': 4.70811744386874, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '27016', 'license': 'MIT', 'n_tokens_mistral': 1014, 'n_tokens_neox': 948, 'n_words': 375, 'path': 'reference/datasets/01-create-dataset-challenge.ipynb', 'repo_name': 'hackalog/easydata-tutorial', 'size': '3304'}