<jupyter_start><jupyter_text>Create pickled dataframes to use for the RAPA demo Data is from the GEO and uses files ```GSE128235_matrix_normalized.txt``` for the b-values and ```GSE125105_series_matrix.txt``` for the meta data. Data can be downloaded with these links:  GSE128235_matrix_normalized.txt GSE125105_series_matrix.txt Note: Files are downloaded gzipped and are downloaded in the ```../data/``` folder. Data summary (from GEO):Genome wide DNA methylation profiling of depressed (n=324) and control subjects (n=209) that were recruited at the Max Planck Institute of Psychiatry (MPIP). The Illumina Infinium 450k Human DNA methylation Beadchip was used to obtain DNA methylation profiles based whole blood samples.  Imports<jupyter_code>import pandas as pd, requests
import gzip, os<jupyter_output><empty_output><jupyter_text>Data dir creation and data download from GEO ftp<jupyter_code>#Make data folder and download b_values and meta data
local_path = "../data/"
if "data" not in os.listdir(".."):
    os.mkdir(local_path)

#If the file is already in the data folder, then skip
name_matrix_norm = "GSE128235_matrix_normalized"
name_series_matrix = "GSE125105_series_matrix"
if "GSE128235_matrix_normalized.txt.gz" not in os.listdir(local_path):
    print(f"Downloading {name_matrix_norm}...")
    GSE128235_matrix_normalized = requests.get("https://ftp.ncbi.nlm.nih.gov/geo/series/GSE128nnn/GSE128235/suppl/GSE128235_matrix_normalized.txt.gz")
    open(local_path + "GSE128235_matrix_normalized.txt.gz", 'wb').write(GSE128235_matrix_normalized.content)
else:
    print(f"Skipping {name_matrix_norm} download (it is already in the data folder)...")

if "GSE125105_series_matrix.txt.gz" not in os.listdir(local_path):
    print(f"Downloading {name_series_matrix}...")
    GSE125105_series_matrix = requests.get("https://ftp.ncbi.nlm.nih.gov/geo/series/GSE125nnn/GSE125105/matrix/GSE125105_series_matrix.txt.gz")
    open(f"{local_path}GSE125105_series_matrix.txt.gz", 'wb').write(GSE125105_series_matrix.content)
else:
    print(f"Skipping {name_series_matrix} download (it is already in the data folder)...")

print(f"{name_matrix_norm} and {name_series_matrix} are in {local_path}")<jupyter_output>Skipping GSE128235_matrix_normalized download (it is already in the data folder)...
Skipping GSE125105_series_matrix download (it is already in the data folder)...
GSE128235_matrix_normalized and GSE125105_series_matrix are in ./data/<jupyter_text>Prepare beta-value pickle<jupyter_code>beta_values = pd.read_csv(f"{local_path}GSE128235_matrix_normalized.txt.gz", sep="\t")
beta_values.head()
beta_values.drop(beta_values.filter(regex=r"(sample\d+_\w+)").columns, axis=1, inplace=True)
beta_values.head()
beta_values = beta_values.set_index(beta_values["ID_REF"]).drop(columns=["Unnamed: 0", "ID_REF"])
beta_values = beta_values.transpose()
print(beta_values.shape)
beta_values.head()
beta_values.to_csv(f"{local_path}GSE128235_published_beta_values.csv")
beta_values.to_pickle(f"{local_path}GSE128235_published_beta_values.pkl")<jupyter_output><empty_output><jupyter_text>Prepare meta-data/targets pickle<jupyter_code>meta_data_lines = gzip.open(f"{local_path}GSE125105_series_matrix.txt.gz").readlines()
meta_data_lines = [str(x) for x in meta_data_lines]
#Parse txt file for diagnosis paired with sample_name
sample_names_in_order, sample_depression_in_order, subset_sample_names, subset_sample_depression = [], [], [], []
for sample in meta_data_lines[25].split("\\t"):
    if sample.startswith("\"genomic"):
        sample_names_in_order.append(sample.split(" ")[3].strip("\"\\\\n\\\'"))
for sample_depression in meta_data_lines[34].split("\\t"):
    if sample_depression.startswith("\"d"):
        sample_depression_in_order.append(sample_depression.split(" ")[1].strip("\"\\\\n\\\'"))
for sample_name, sample_depression in zip(sample_names_in_order, sample_depression_in_order):
    if sample_name in beta_values.index:
        subset_sample_names.append(sample_name)
        subset_sample_depression.append(sample_depression)
targets = pd.DataFrame(subset_sample_depression, index=pd.Index(subset_sample_names, name="Sample_Name"), columns=["diagnosis"]).sort_index()
targets.to_csv(f"{local_path}GSE128235_GPL13534_meta_data.csv")
targets.to_pickle(f"{local_path}GSE128235_GPL13534_meta_data.pkl")
targets<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11940298507462686, 'fraction_numerical': 0.046153846153846156, 'mean_word_length': 6.934426229508197, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '529302', 'license': 'MIT', 'n_tokens_mistral': 1657, 'n_tokens_neox': 1494, 'n_words': 292, 'path': 'tutorials/00-data-preparation.ipynb', 'repo_name': 'FOXOBioScience/rapa', 'size': '4355'}