<jupyter_start><jupyter_text>Pixel analysis part 1 Download datasets from Metaspace  Set paths Input`metadata_path`: metadata table (I prefer downloaded from Metaspace)  Needs to have the following columns:  `datasetId` - Metaspace id  `datasetName`	- name of the dataset  All other columns are optional, generally useful to have "condition" column or something similar used for comparison in the next steps of analysis  Script will add to the metadata table additional column "data_file" and save updated metadata to the analysis directory at `metadata_save_path` Output`data_dir`: general root dir for storing data, including downloaded Metaspace datasets and plots  `metaspace_data_dir`: directory, where ion images and metadata will be stored`plots_path`: directory for plots  (ion images, cell masks, plots from scanpy analysis)<jupyter_code>data_dir = Path(r"/Users/alberto-mac/EMBL_ATeam/projects/gastrosome_compare_conditions")
metaspace_data_dir = data_dir / "pixel_analysis_reannotated" / "metaspace_data"
metaspace_data_dir.mkdir(parents=True, exist_ok=True)

metadata_path = Path(r"/Users/alberto-mac/EMBL_ATeam/projects/gastrosome_compare_conditions/metaspace_datasets.csv")
metadata_save_path = data_dir / "pixel_analysis_reannotated" / "metaspace_datasets_paths.csv"

plots_path = data_dir / "pixel_analysis_reannotated"/ "plots"
plots_path.mkdir(parents=True, exist_ok=True)<jupyter_output><empty_output><jupyter_text>Check and subset metadata Metadata file downloaded from Metaspace will have the two line header, then column names and then rows corresponding to the datasets. I prefer to keep the header just in case and read in with pandas starting with the third line<jupyter_code>metadata = pd.read_csv(metadata_path, skiprows=2)
metadata<jupyter_output><empty_output><jupyter_text>Doing analysis with all samples<jupyter_code>subset_metadata = metadata<jupyter_output><empty_output><jupyter_text>Set up Metaspace download parameters<jupyter_code># Molecules Databases to download:
databases = [
    ['Gastrosome_pixel_analysis_intra_ions', 'v1'],
    # ["SwissLipids", "2018-02-02"]
]

# Maximum false discovery rate. Valid values are 0.05, 0.1, 0.2, 0.5 corresponding to 5%, 10%, 20% and 50%
fdr = 0.5<jupyter_output><empty_output><jupyter_text>Set paths of the datasets and store final metadata table<jupyter_code>subset_metadata.loc[:, "metaspace_download_dir_path"] = [str(metaspace_data_dir / "slide_{slide}/W{well}".format(**ds_info.to_dict())) for _, ds_info in subset_metadata.iterrows()]

subset_metadata.to_csv(metadata_save_path, index=False)<jupyter_output><empty_output><jupyter_text>Download- You need your METASPACE credentials set in your `~/.metaspacecredentials` file(which should be already in place if you already used SpaceM).- Next, create a SpaceM config file and select the molecules databases you want to download in the `Mass Spectrometry Annotation` step<jupyter_code># Create a temporary SpaceM config with infos about the molecule databases that you want to download:
# path_spacem_config = "/Users/alberto-mac/EMBL_ATeam/projects/gastrosome_compare_conditions/spacem_config_template.json"
# spacem_config = read_project_config(path_spacem_config)

# Create empty SpaceM config and specify the needed molecule databases
spacem_config = ProjectConfig()
spacem_config.mass_spec_annotation.metaspace.databases = databases
spacem_config.mass_spec_annotation.metaspace.fdr = fdr



# Silence some warning while saving adata:
import warnings
warnings.filterwarnings("ignore", category=FutureWarning, message=r".*Reordering categories will always return a new Categorical object.*")
warnings.filterwarnings("ignore", category=FutureWarning, message=r".*is_categorical is deprecated and will be removed in a future version.*")
# warnings.filterwarnings("ignore", category=FutureWarning, message=r".*Transforming to str index.*")


# Create adata file:
# ds_meta = subset_metadata[0]
for id, ds_meta in subset_metadata.iterrows():
    print(f"Downloading data for {ds_meta.datasetName}")
    dataset_id = ds_meta["datasetId"]
    metaspace_download_dir_path = ds_meta["metaspace_download_dir_path"]
    os.makedirs(metaspace_download_dir_path, exist_ok=True)

    # Manually set the dataset ID in the config object:
    spacem_config.mass_spec_annotation.metaspace.dataset_id = dataset_id

    # Download from METASPACE:
    annotation_loader = MetaSpaceAnnotationsLoader(output_dir=Path(metaspace_download_dir_path),
                               config=spacem_config)
    ion_images = annotation_loader.run().ion_images

    # Save metadata to disk:
    ion_metadata = ion_images.metadata
    ion_images_metadata_path = Path(metaspace_download_dir_path) / "ion_images_metadata.csv"
    write_ion_metadata(ion_metadata, ion_images_metadata_path)

    cell_mask = np.ones(ion_images.shape, dtype="bool")

    cell_indices = cell_mask.nonzero()

    count_matrix = np.reshape(ion_images.array, (ion_images.array.shape[0], 1, -1))

    count_matrix = count_matrix[:, cell_mask.reshape((1, -1)) > 0]
    obs_df = pd.DataFrame(data={"x": cell_indices[0], "y": cell_indices[1]})
    for col in ds_meta.index:
        obs_df[col] = ds_meta[col]

    # # Add TIC to .obs
    # obs_df["tic"] = np.reshape(tic, (1, -1))[cell_mask.reshape((1, -1)) > 0]

    # Fix some types to avoid problems while saving adata:
    ion_metadata.databases = ion_metadata.databases.astype("str")
    ion_metadata.moleculeIds = ion_metadata.moleculeIds.astype("str")
    ion_metadata.moleculeNames = [str(name) for name in ion_metadata.moleculeNames]
    ion_metadata.fdr = ion_metadata.fdr.astype("str")

    adata = sc.AnnData(X=count_matrix.T,
                       obs=obs_df,
                       var=ion_metadata.set_index("annotation_id"))

    adata.write(Path(metaspace_download_dir_path) / "adata.h5ad")


ds_meta.datasetName<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09144994050654429, 'fraction_numerical': 0.007479177290498045, 'mean_word_length': 5.709236031927024, 'pattern_counts': {'":': 2, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '353650', 'license': 'MIT', 'n_tokens_mistral': 1919, 'n_tokens_neox': 1828, 'n_words': 459, 'path': 'projects/gastrosome_compare_conditions/pixel_analysis/03_redownload_datasets.ipynb', 'repo_name': 'abailoni/single-cell-analysis', 'size': '5883'}