<jupyter_start><jupyter_text>test of using Pandas DataFrame as internal structureAdvantages* well supported* has connection to lots of I/O layers (even PyTables)* can add/remove columns easilyProblems:* can't seem to store vector data in a column...<jupyter_code>import pandas as pd
import numpy as np
evid = np.arange(10) + 1024
X = np.linspace(-10,10,10, dtype=np.float32)
Y = np.linspace(-10,10,10, dtype=np.float32)
V = np.random.uniform(size=10)
data = pd.DataFrame({ 'DETX':X,'DETY':Y,'VALUE':V}, index=evid)
data
data.describe()
print data.loc[1029]
for col in data:
    print col<jupyter_output>DETX
DETY
VALUE<jupyter_text>Adding a column:<jupyter_code>energy = np.random.lognormal(2,1,size=10)
data['ENERGY'] = energy
data<jupyter_output><empty_output><jupyter_text>adding array data:<jupyter_code>pixval = np.arange(5*10).reshape(10,5)
print pixval
try:
    data['PIXVAL']=pixval
except Exception,e:
    print "FAILED:",e
data<jupyter_output>[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]
 [15 16 17 18 19]
 [20 21 22 23 24]
 [25 26 27 28 29]
 [30 31 32 33 34]
 [35 36 37 38 39]
 [40 41 42 43 44]
 [45 46 47 48 49]]
FAILED: Wrong number of items passed 5, placement implies 1
{'fraction_non_alphanumeric': 0.12740989103101424, 'fraction_numerical': 0.10896898575020955, 'mean_word_length': 4.852941176470588, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '6116', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 563, 'n_tokens_neox': 463, 'n_words': 144, 'path': 'examples/brainstorm/notebooks/pandas dataframes.ipynb', 'repo_name': 'rdelosreyes/myctapipe', 'size': '1193'}