<jupyter_start><jupyter_text>Tech A and Tech CTech A and C were collected as a bunch of software-timed measurements. This part just selects a few cells to reduce the data size.<jupyter_code># Load tech A or tech C time series data
colnames = ["addr", "time", "r", "g"]
tsdata = pd.read_csv(f"tsdata{TECH}.in.tsv.gz", names=colnames, sep='\t')
# Cleaner format
# range_index TAB g0 TAB g1 TAB g2 ...
with open(f"../data/tech{TECH}/tsdata.tsv", "w") as outf:
    for r, (addr, ts) in enumerate(tsdata.groupby("addr", sort=False)["g"]):
        outf.write(f"{r % 32}\t0\t")
        np.savetxt(outf, np.expand_dims(ts, 0), delimiter='\t')
# Same cleaner format for subset of ranges we want to use
indices = sorted([r+32*o for r, o in zip(settings["ts_ranges"], settings["ts_index_offsets"])])
with open(f"../data/tech{TECH}/tsdata.min.tsv", "w") as outf:
    for i, (addr, ts) in enumerate(tsdata.groupby("addr", sort=False)["g"]):
        if i not in indices:
            continue
        outf.write(f"{i % 32}\t0\t")
        np.savetxt(outf, np.expand_dims(ts, 0), delimiter='\t')<jupyter_output><empty_output><jupyter_text>Tech BTech B was collected with the high-frequency scope so it is in a different format the Techs A and C. Here, we simply filter the data down to a few ranges we care about.<jupyter_code># Get time series data for tech B
waveforms = []
with open("tsdataB.in.tsv") as fp:
    for i, line in enumerate(fp):
        if i in settings["ts_ranges"]:
            data = np.array(line.strip().split('\t'), dtype=np.float32)
            waveforms.append(data)
# Output results for tech B
waveforms = np.array(waveforms)
np.savetxt("../data/techB/tsdata.min.tsv.gz", waveforms, delimiter='\t')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15092165898617513, 'fraction_numerical': 0.008640552995391706, 'mean_word_length': 3.8929577464788734, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '167026', 'license': 'MIT', 'n_tokens_mistral': 638, 'n_tokens_neox': 619, 'n_words': 185, 'path': 'preprocessing/tsdata_conversion.ipynb', 'repo_name': 'akashlevy/RRAM-Relaxation-Master-Model', 'size': '1736'}