<jupyter_start><jupyter_text>GenoSurf API Example Use Case:  Extract TCGA pairs of items aligned to hg19 legacy assembly and to new GRCh38 assembly (from Genomic Data Commons portal) Import necessary libraries<jupyter_code>import requests
from ast import literal_eval
import pandas as pd
from functools import reduce<jupyter_output><empty_output><jupyter_text>Call API with queryThe ```data``` variable contains the GenoSurf query in JSON format (it can be downloaded from the interface clicking on MODIFY, then 'COPY TO CLIPBOARD').The ```payload``` variable contains the JSON payload of the API POST request. For advanced use refer to the first notebook (GenoSurf_rest_ENCODE_assemblies.ipynb).<jupyter_code>url = 'http://geco.deib.polimi.it/genosurf/api/query/table'

data = '{"gcm":{"project_name":["tcga-chol"]},"type":"original","kv":{}}'

data_dict = literal_eval(data)

payload = {'agg': True}<jupyter_output><empty_output><jupyter_text>Load result dataset into Pandas DataFrameUse ```len(response_json)``` to visualize the number of extracted items.Use ```df.shape[1]``` to visualize the number of columnsUse ```df.columns``` for the complete list of columns<jupyter_code>response = requests.post(url, json=data_dict, params=payload)
response_json = response.json()

df = pd.DataFrame(response_json)

#restrict the dataframe to only interesting columns
df = df[['item_source_id','alt_item_source_id','local_url','source_url', 'assembly','data_type']]<jupyter_output><empty_output><jupyter_text>**Define the role of columns***   ```class_column```: used for distinguishing data in classes*   ```grouped_columns```: used for grouping*   ```other_columns```: all columns that are not used for grouping<jupyter_code>class_column = 'assembly'
grouped_columns = ['alt_item_source_id']
other_columns = list(filter(lambda x: not x in grouped_columns, df.columns))<jupyter_output><empty_output><jupyter_text>**Compute desider output**Use ```full_result.head(10)``` to visualize first rows of the final dataframe.In the dataframe:*   First columns are from ```grouped_columns```*   Then we have columns used for distinguishing classes (column names are formed as ```column_name_``` to separate classes, e.g. ```assembly_GRCh38``` and ```assembly_hg19```)<jupyter_code>df = df[grouped_columns + other_columns]

#define possible values of class column
classes = df[class_column].sort_values().unique()

#perform group by on the dataframe
grouped_df = df.groupby(grouped_columns)

result = []

#compute the pairs
for group_key, group_df in grouped_df:

  inner_grouped_df = group_df.groupby(class_column)
  
  has_all_classes = reduce((lambda x, y: x and y), map(lambda x: x in inner_grouped_df.groups.keys(), classes))
    
  if has_all_classes:
    class_result = []
    for c in classes:
      inner_group_df = inner_grouped_df.get_group(c) 
      inner_group_df.columns = map(lambda x: x if x in grouped_columns else x + "_" + c.lower() ,inner_group_df.columns)
      class_result.append(inner_group_df)
    
    merged = reduce(lambda left,right: pd.merge(left,right,on=grouped_columns), class_result)

    result.append(merged)

#builds the whole final dataframe result    
full_result = pd.concat(result, axis=0, join='outer')

print("Number of pairs: ",full_result.shape[0])

full_result.head(10)<jupyter_output>Number of pairs:  573<jupyter_text>**Visualize and save the list of pairs**If you are interested only in some of the the pairs list, extract if from the ```full_result``` and exploit it further.<jupyter_code>pairs = full_result[['alt_item_source_id','local_url_grch38','local_url_hg19']]

pairs.to_csv("GenoSurf_rest_TCGA_assemblies.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11427795874049945, 'fraction_numerical': 0.005971769815418024, 'mean_word_length': 6.566735112936345, 'pattern_counts': {'":': 4, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '418871', 'license': 'Apache-2.0', 'n_tokens_mistral': 1207, 'n_tokens_neox': 1164, 'n_words': 329, 'path': 'backend/examples/GenoSurf_rest_TCGA_assemblies.ipynb', 'repo_name': 'LucaCilibrasi/docker_viruclust', 'size': '3684'}