<jupyter_start><jupyter_text>Our data generator will have three components. First, we define the link between the "magic" integer ids and the object attributes (e.g. name, location, etc. of activities in EXIOBASE). This will be in a Pandas dataframe.<jupyter_code>import pandas as pd
df = pd.DataFrame([
    {'index': 0, 'name': 'foo', 'location': 'CH'},
    {'index': 1, 'name': 'foo', 'location': 'FR'},
    {'index': 2, 'name': 'bar', 'location': 'DE'},
]).set_index('index')
df<jupyter_output><empty_output><jupyter_text>Then we have the generator function itself. This can generate an infinite series, but we need to define ahead of time the number of rows (i.e. the number of exchanges that will be modified).We will have two rows in this example. When iterated over, our function will therefore return a 1-d numpy array of length 2.<jupyter_code>import numpy as np
def gen_func():
    while True:
        yield np.random.random(size=(2,))
for x, y in zip(range(10), gen_func()):
    print(x, y)<jupyter_output>0 [0.80917835 0.18277058]
1 [0.99828613 0.94558437]
2 [0.14162607 0.23392832]
3 [0.02861499 0.92182092]
4 [0.47642805 0.96394575]
5 [0.81737114 0.57888545]
6 [0.10742288 0.76787843]
7 [0.18578976 0.41947305]
8 [0.8404466 0.4388719]
9 [0.21330664 0.1809946 ]<jupyter_text>Finally, we need to define which exchange both elements in our data generator refer to. Here, we again use our "magic" integer ids.<jupyter_code>exchanges = [
    {"row": 0, "col": 1},
    {"row": 1, "col": 2},
]<jupyter_output><empty_output><jupyter_text>We can now add all three components to a processed data package.<jupyter_code>from bw_processing import *
dirname = "generator_metadata"<jupyter_output><empty_output><jupyter_text>This will create the directory ``dirname`` in the working directory of this notebook. You can also pass in a string (or `pathlib.Path` instance) for any other path.If you run this twice, you will need to change `dirname` or delete the created directory, or you will get an error.<jupyter_code>dp = Datapackage.create(dirname)<jupyter_output><empty_output><jupyter_text>The order things are added in is important, and we need to include the logical connection between them.<jupyter_code>dp.add_presamples_data_array(
    gen_func,
    matrix_label="technosphere", 
    name="infinite-data", 
    is_interface=True # The magic that allows for infinite sequences
)
dp.add_presamples_indices_array(
    # indices_wrapper allows us to use a list of dicts, 
    # instead of building the array ourselfes
    indices_wrapper(exchanges), 
    data_array="infinite-data",
    name="infinite-data-indices",
    nrows=2
)
dp.add_csv_metadata(
    df, 
    valid_for=[("infinite-data-indices", "rows"), ("infinite-data-indices", "cols")], 
    name="infinite-data-meta"
)
dp.metadata
dp.data
dp.finalize()
dp.data
dp.resources
ls generator_metadata/<jupyter_output>datapackage.json           infinite-data-meta.csv
infinite-data-indices.npy<jupyter_text>Test loading to make sure things are reasonable.<jupyter_code>from bw_processing import *
dp = Datapackage.load("generator_metadata")
dp.data
dp.metadata<jupyter_output><empty_output><jupyter_text>To make this fully functional, we just need to add the generator data resource with the same name.<jupyter_code>dp.add_presamples_data_array(
    gen_func,
    matrix_label="technosphere", 
    name="infinite-data", 
    is_interface=True
)
dp.metadata<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1105629715612304, 'fraction_numerical': 0.05803830528148578, 'mean_word_length': 5.005226480836237, 'pattern_counts': {'":': 4, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '58871', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1264, 'n_tokens_neox': 1136, 'n_words': 356, 'path': 'dev/Presamples generator example.ipynb', 'repo_name': 'ecoinvent/bw_processing', 'size': '3446'}