<jupyter_start><jupyter_text>For the velocity analysis lowly expressed (spliced) genes were excluded (requiring 40 minimum expressed counts and detected over 30 cells) and the top 3,000 high variable genes were selected on the basis of a non-parametric fit of the coefficient of variation (CV) using the mean as predictor (using support vector regression). Only 1,706 genes that had unspliced molecule counts above a detection threshold (25 minimum expressed counts and detected over 20 cells) were kept for the analysis. To normalize for the cell size, the counts were divided by the total number of molecules in each cell, and multiplied by the mean number of molecules across all cells. Spliced and unspliced counts were normalized separately. To reduce dimensionality, PCA was performed and the top 19 variable components were kept on the basis of the explained variance ratio profile. Euclidean distance in this reduced PCA space was used to construct a kNN graph (k = 500), using a greedy balanced kNN algorithm that limits each node to a maximum of 4k incoming edges. This graph was used to perform kNN pooling. Velocity-based extrapolation was performed using model I assumptions.<jupyter_code>spliced.shape
unspliced.shape
spliced[1,:] > 10
# sum(spliced[30000,:] > 40)
counter = 0
for i in range(32738):
    if sum(spliced[i,:] >= 5):
        print(i)
        counter += 1
counter = 0
for i in range(32738):
    if sum(spliced[i,:] >= 5) >= 20:
        print(i)
        counter += 1
counter
# spliced[1,:].shape
sum(spliced[23,:] >= 5) >= 20
print(counter)
def gene_expr_filter(data, gene_min_read=20, min_cell_count=30):
    '''
    Filter for genes that are above a certain
    read count cutoff, and above a certain cell coun
    '''
    geneindex = []
    genecount = data.shape[0]
    for i in range(genecount):
        cells_with_min_read_at_gene = sum(data[i,:] >= gene_min_read)
        if cells_with_min_read_at_gene >= min_cell_count:
            geneindex.append(i)
    
    return geneindex


spliced_gene_index = gene_expr_filter(spliced, gene_min_read=5, min_cell_count=20)
    
    
len(spliced_gene_index)
unspliced_gene_index = gene_expr_filter(unspliced, gene_min_read=3, min_cell_count=20)
len(unspliced_gene_index )
set(spliced_gene_index) in set(spliced_gene_index)
genes_union = set(spliced_gene_index).union(set(unspliced_gene_index))
genes_intersect = set(spliced_gene_index).intersection(set(unspliced_gene_index))
# Normalize gene expression
# len(genes_intersect)

spliced_normcell = spliced / np.sum(spliced, axis=0)
unspliced_normcell = unspliced / np.sum(unspliced, axis=0)

# np.sum(unspliced, axis=0)
spliced_filter = spliced[list(genes_intersect),:]
unspliced_filter = unspliced[list(genes_intersect),:]
spliced_gene_index
list(genes_intersect)
unspliced.shape
x = np.array([[20,20,20],[30,30,30],[40,40,40]])
x
v = np.array([20,30,40])
x.T / v
spliced_normcell_filter = spliced_normcell[list(genes_intersect),:]
unspliced_normcell_filter = unspliced_normcell[list(genes_intersect),:]
# spliced_normcell_filter.append(unspliced_normcell_filter)

normcell_filter_stack = np.stack((spliced_normcell_filter, unspliced_normcell_filter))
normcell_filter_stack[0,:,:]
normcell_filter_stack.dump("norm_filtered_cells.pickle")
normcell_filter_stack.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08858089786080145, 'fraction_numerical': 0.02862307924073516, 'mean_word_length': 5.509803921568627, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '659291', 'license': 'MIT', 'n_tokens_mistral': 1191, 'n_tokens_neox': 1069, 'n_words': 315, 'path': 'rnavelocity/processed_data/Clean_and_normalize.ipynb', 'repo_name': 'Topcoder-Kitty-ML/cs207-FinalProject', 'size': '3319'}