<jupyter_start><jupyter_text>Single-Cell RNA-seq Clustering Analysis Notebook**Author** - <NAME>**Email** - <EMAIL>This notebook analyzes a dataset of [3K Peripheral Blood Mononuclear Cells (PBMCs) from a Healthy Donor](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k) available from 10X Genomics, sequenced on the Illumina NextSeq 500. Steps are modeled after the [Seurat Guided Clustering Tutorial](http://satijalab.org/seurat/pbmc3k_tutorial.html) using the [scanpy](https://github.com/theislab/scanpy) library.  ObjectiveThe goal of this notebook is to provide a standard single-cell RNA-seq analysis workflow for pre-processing, identifying sub-populations of cells by clustering, and exploring biomarkers to explain intra-population heterogeneity. DatasetThe example dataset consists of the expression of 2,700 single PBMCs. Analysis Overview1. [**Setup Analysis**](Step-1:-Setup-Analysis)    1. Load raw count matrix.2. [**Preprocess Counts**](Step-2:-Preprocess-Counts)    1. Filter cells based on QC metrics.    2. Perform data normalization and scaling.    3. Remove unwanted sources of variation (number of detected molecules per cell as well as the percentage mitochondrial gene content).    4. Detect highly variable genes.    5. Perform linear dimensional reduction (PCA).3. [**Cluster Cells**](Step-3:-Cluster-Cells)    1. Cluster cells (graph-based clustering) in PCA space and visualize using t-SNE.4. [**Visualize Cluster Markers**](Step-4:-Visualize-Cluster-Markers)    1. Explore and visualize cluster markers interactively.5. [**Export Analysis Data**](Step-5:-Export-Analysis-Data)    1. Export data to `.csv` files or a compressed `.h5ad` format.  Step 1: Setup AnalysisLoad a raw count matrix for a single-cell RNA-seq experiment.  Instructions Provide your data file either as a URL or local file path. Select "2700 PBMCs from a Healthy Donor (example)" from the dropdown menu to load the example dataset.Supported file formats: `csv`, `xlsx`, `txt`, `tsv`, `tab`, `data`, `h5`, `h5ad`, `soft.gz`, `txt.gz`, `anndata`, `mtx*`Text and Excel files (csv, txt, tsv, tab, data, xlsx): Gene and sample names are assumed to be the first column and row respectively.NOTE*: The 10x Genomics genomics pipeline generates gene-barcode matrices usually named `matrix.mtx`, `genes.tsv`, and `barcodes.tsv`. If the `mtx` files is provided, the genes and barcodes files will autoomatically be imported from the same folder.<jupyter_code>import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

import sys
sys.path.append('.')

import genepattern
from singlecell import SingleCellAnalysis, sc
import logging
sc.settings.verbosity = 0
logging.disable(logging.INFO)

%matplotlib inline

analysis = SingleCellAnalysis()
genepattern.GPUIBuilder(
    analysis.setup_analysis,
    function_import='analysis.setup_analysis',
    name='Setup Analysis',
    parameters={
        'matrix_filepath': {
            'name': 'matrix data file',
            'description': 'Provide your data file either as a URL or local file path. See above instructions for supported formats.',
            'default': '2700 PBMCs from a Healthy Donor (example)',
            'type': 'file',
            'choices': {'2700 PBMCs from a Healthy Donor (example)': 'https://github.com/genepattern/seurat_python_notebook/raw/master/data/matrix.mtx'},
            'kinds': ['csv', 'xlsx', 'txt', 'tsv', 'tab', 'data', 'h5', 'h5ad', 'soft.gz', 'txt.gz', 'anndata', 'mtx'],
        },
        'output_var': {
            'hide': True
        }
    }
)<jupyter_output><empty_output><jupyter_text>Step 2: Preprocess CountsPerform cell quality control by evaluating quality metrics, normalizing counts, scaling, and correcting for effects of total counts per cell and the percentage of mitochondrial genes expressed. Then detect highly variable genes and perform linear dimensional reduction (PCA).Instructions Use the quality metrics displayed in the output of **Step 1** to detect outlier cells and set ranges to filter them. Removing unwanted sources of variation may take some time to complete.<jupyter_code>genepattern.GPUIBuilder(
    analysis.preprocess_counts,
    function_import='analysis.preprocess_counts',
    name='Preprocess Counts',
    parameters={
        'data': {
            'description': 'Output from the "Setup Analysis" tool.',
            'default': 'analysis'
        },
        
        'min_n_cells': {
            'name': 'filter genes (min. # of cells)',
            'description': 'Include genes expressed in at least this many cells. Blank will be treated as 0.',
            'type': 'number',
            'default': 3
        },
        
        'min_n_genes': {
            'name': 'filter cells (min. # of genes)',
            'description': 'Include cells with at least this many genes. Blank will be treated as 0.',
            'type': 'number',
            'default': 200
        },
        'max_n_genes': {
            'name': 'filter cells (max # of genes)',
            'description': 'Include cells with at most this many genes. Blank will be treated as no maximum value.',
            'type': 'number',
            'default': 2500
        },
        'min_n_counts': {
            'name': 'filter cells (min. total counts)',
            'description': 'Include cells with at least this many counts. Blank will be treated as 0.',
            'type': 'number',
            'default': 0
        },
        'max_n_counts': {
            'name': 'filter cells (max total counts)',
            'description': 'Include cells with at most this many counts. Blank will be treated as no maximum value.',
            'type': 'number',
            'default': 15000
        },

        'min_percent_mito': {
            'name': 'filter cells (min. % mito. genes)',
            'description': 'Include cells with at least this % of genes that are mitochondrial genes. Blank will be treated as 0.',
            'type': 'number',
            'default': 0
        },
        'max_percent_mito': {
            'name': 'filter cells (max % mito. genes)',
            'description': 'Include cells with at most this % of genes that are mitochondrial genes. Blank will be treated as no maximum value.',
            'type': 'number',
            'default': 15
        },
        'normalization_method': {
            'name': 'log normalize',
            'description': 'Perform log normalization on the data.',
            'choices': {'Yes': 'LogNormalize', 'No': ''}
        },
        'output_var': {
            'hide': True
        }
    })<jupyter_output><empty_output><jupyter_text>Step 3: Cluster CellsCluster cells using the Louvain method for community detection on the top  of principal components. Then use t-SNE to visualize cells, again using the top  of principal components.Instructions Perform clustering and visualization of the cells using the sliders to tune parameters. Look at the plot in **Step 2** showing the standard deviations of the principle components and draw a cutoff where there is a clear elbow in the graph. The components to the left of this cutoff are then used to cluster the cells.<jupyter_code>genepattern.GPUIBuilder(
    analysis.cluster_cells,
    function_import='analysis.cluster_cells',
    name='Open Cell Clustering Interface',
    description='This step outputs an interactive interface to cluster cells.',
    parameters={
        'data': {
            'name': 'analysis object',
            'description': 'Use the output variable from "Step 2".',
            'default': 'analysis'
        },
        'pcs': {
            'name': '# of principal components',
            'description': 'The number of principal components to use to cluster cells. Determine the number of principal components (PCs) to use by drawing a cutoff where there is a clear elbow in the graph above.'
        },
        'resolution': {
            'description': 'Higher resolution means more and smaller clusters. We find that values 0.6-1.2 typically returns good results for single cell datasets of around 3K cells. Optimal resolution often increases for larger datasets.'
        },
        'perplexity': {
            'description': 'The perplexity parameter loosely models the number of close neighbors each point has.'
        },
        'output_var': {
            'hide': True
        }
    })<jupyter_output><empty_output><jupyter_text>Step 4: Visualize Cluster MarkersVisualization the expression of markers on the clustering plot. Explore genes that are differentially expressed expessed in clusters as potential biomarkers.Instructions The following are canonical markers that mark known cell types in this dataset. These can be used to identify what cell types are present and how they correspond to clusters.| Cell Type | Markers || --------- | ------- || CD4 T Cells | IL7R || LYZCD14+ Monocytes | CD14, LYZ || B cells | MS4A1 || CD8 T cells | CD8A || FCGR3A+ Monocytes | FCGR3A, MS4A7 || NK cells | GNLY, NKG7 || Dendritic Cells | FCER1A, CST3 || Megakaryocytes | PPBP |<jupyter_code>genepattern.GPUIBuilder(
    analysis.visualize_markers,
    function_import='analysis.visualize_markers',
    name='Open Visualize Cluster Markers Interface',
    description='This step outputs an interactive interface to explore gene expression in clusters of cells.',
    parameters={
        'output_var': {
            'hide': True
        }
    }
)<jupyter_output><empty_output><jupyter_text>Step 5: Export Analysis DataInstructions Using the cell below, export results as a series of .csv files or compressed .h5ad file.Relevant Files    X.csv - The preprocessed expression matrix of cells (rows) by genes (columns). This only includes variable genes, usually a much smaller subset of genes compared to the raw counts.    obs.csv - Cell annotations including  of genes, % mitochondrial genes,  and cluster assignments    obsm.csv - Coordinates of cells in various dimensional reduction spaces (e.g., PCA, t-SNE).    var.csv - Gene annotations (of variable genes) including the  of cells, mean expression, dispersion, and normalized dispersion statistics.    varm.csv - Loadings of cells in various dimensional reduction spaces (e.g., PCA, t-SNE).    uns/pca_variance_ratio.csv - % variance explained by each principal component.    uns/rank_genes_groups_gene_names.csv - Names of the top ranked marker genes for each cluster.    uns/rank_genes_groups_gene_scores.csv - z-scores of the top ranked marker genes for each cluster.<jupyter_code>genepattern.GPUIBuilder(
    analysis.export_data,
    function_import='analysis.export_data',
    name='Export Analysis Data',
    description='Export data as a series of .csv files or compressed .hda5 file.',
    parameters={
        'path': {
            'name': 'filepath',
            'description': 'Name of directory where file(s) will be saved. Exporting as an h5ad file produces a single file output.',
            'default': 'data/analysis'
        },
        'h5ad': {
            'name': 'file format',
            'description': 'Choose to save either as .csv files or as a single compressed .h5ad-formatted hdf5 file. It is recommended to export .csv files unless you know what you are doing.',
            'default': False,
            'choices': {
                'Comma-separated values (.csv)': False,
                'H5AD file (HDF5 file in the AnnData formatting convention)': True
            },
        },
        'output_var': {
            'hide': True
        }
    }
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09863916096038831, 'fraction_numerical': 0.009967929271040999, 'mean_word_length': 3.0669721536834684, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '219015', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3220, 'n_tokens_neox': 3027, 'n_words': 1229, 'path': 'Single-Cell RNA-seq Clustering Analysis Notebook.ipynb', 'repo_name': 'bahill/seurat_python_notebook', 'size': '11550'}