<jupyter_start><jupyter_text>Explore table of raw expression from BY4741 yscRNA-seq libraries This notebook is meant to facilitate exploring the data table of raw expression from BY4741 yscRNA-seq libraries (Supplementary Table 3 of [Nadal-Ribelles et al., 2019](https://www.ncbi.nlm.nih.gov/pubmed/30718850)) interactively.Reference for the data:  - [Sensitive high-throughput single-cell RNA-seq reveals within-clonal transcript correlations in yeast populations. Nadal-Ribelles M, Islam S, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>M. Nat Microbiol. 2019 Feb 4. doi: 10.1038/s41564-018-0346-9. Epub ahead of print. PMID: 30718850 ](https://www.ncbi.nlm.nih.gov/pubmed/30718850)**Technical note: Presently this notebook will only fully work in the classic Jupyter notebook environment when using via MyBinder.org. I don't have the repo set up to use JupyterLab with qgrid as of yet.** -----If you haven't used one of these notebooks before, they're basically web pages in which you can write, edit, and run live code. They're meant to encourage experimentation, so don't feel nervous. Just try running a few cells and see what happens!.    Some tips:            Code cells have boxes around them. When you hover over them an  icon appears.        To run a code cell either click the  icon, or click on the cell and then hit Shift+Enter. The Shift+Enter combo will also move you to the next cell, so it's a quick way to work through the notebook.        While a cell is running a * appears in the square brackets next to the cell. Once the cell has finished running the asterix will be replaced with a number.        In most cases you'll want to start from the top of notebook and work your way down running each cell in turn. Later cells might depend on the results of earlier ones.        To edit a code cell, just click on it and type stuff. Remember to run the cell once you've finished editing.    ----  Preparation First get the file associated with supplemental data table 3.<jupyter_code>!curl -OL https://static-content.springer.com/esm/art%3A10.1038%2Fs41564-018-0346-9/MediaObjects/41564_2018_346_MOESM3_ESM.txt<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1964k  100 1964k    0     0  2260k      0 --:--:-- --:--:-- --:--:-- 2257k<jupyter_text>Read in table and begin to evaluate itUnder ['Supplementary information' here](https://www.nature.com/articles/s41564-018-0346-9Sec22), it says:>"Supplementary Table 3Raw expression BY4741 yscRNA-seq libraries after applying the quality filter criteria (total of 127 cells). Table contains raw number of molecules for each gene (rows) for each cell (rows)."(I suspect the latter `(rows)` should read `(columns)`.)After reading in the data, we'll also **add columns** for the `Total` raw expression counts and the average (`AVG`).<jupyter_code>import pandas as pd
df = pd.read_csv('41564_2018_346_MOESM3_ESM.txt', sep='\t', header=1, delim_whitespace=False)
df['Total']=df.sum(1)
df['AVG']=df.mean(1).round(2)
print(f"There is information for {len(df)} genes.")
df.head()<jupyter_output><empty_output><jupyter_text>Using  qgrid for interactive viewingAs the information above showed, there is a lot of data. To make it more intuitive to explore we'll use the `qgrid` package by [Quantopian, Inc.](https://github.com/quantopian).The code in the cell below will bring up a view of the dataframe. Clicking on the column names will allow you to sort them by that parameter. For example, scroll to the right side in the view to show the `Total` column and then click on the column heading twice to sort it descending. You can click the expand button on the right above the column to have the dataframe view fill the current browser window.<jupyter_code>import qgrid
# controlling individual column widths based on https://github.com/quantopian/qgrid/issues/119 ; however if you
# have too many columns, you still need `grid_options={'forceFitColumns': False}` from 'Example 2 - Render a DataFrame with 1 million rows'
col_options = {'width': 55,}
col_defs = {
    'geneName': {'width': 85,},
    'comGeneName':{'width': 100,},
    'Total':{'width': 65,},
    'AVG':{'width': 65,},
}
#qgrid_widget = qgrid.show_grid(df, show_toolbar=True, grid_options={'forceFitColumns': False, 'defaultColumnWidth': 65})
qgrid_widget = qgrid.show_grid(df, column_options=col_options,column_definitions=col_defs, grid_options={'forceFitColumns': False}, show_toolbar=True)
qgrid_widget<jupyter_output><empty_output><jupyter_text>However, that still isn't that easy to explore. Since we have the `Total` and average columns, we could probably now show the results for every cell all the time.<jupyter_code>import qgrid
just_sum_df = df[['geneName','comGeneName','Total','AVG']]
#qgrid_widget = qgrid.show_grid(df, show_toolbar=True, grid_options={'forceFitColumns': False, 'defaultColumnWidth': 65})
qgrid_widget = qgrid.show_grid(just_sum_df, show_toolbar=True)
qgrid_widget<jupyter_output><empty_output><jupyter_text>Locating your favorite genes in the dataUnless we are going to use qgrid to view results, we probably want to sort on the total first so the resulting number on the left when we subset to show our favorite genes corresponds to the ranking for total counts. Otherwise that number will just reflect the genes postion among the each chromosome in order starting with chromosome 1. (That was why earlier when we first ran `df.head()` to look at the top of the dataframe we saw `YAL067C` first.)The next cell will sort the two dataframes we have been using.<jupyter_code>SORTED_df = df.sort_values('Total', ascending=False).reset_index(drop=True) #if we left off `drop=True`, we'd keep the 
# number before sorting on the left as well& it would be under the heading 'index'
just_sum_SORTED_df = just_sum_df.sort_values('Total', ascending=False).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Now to subset.<jupyter_code>SORTED_df[SORTED_df.comGeneName == "POP4"]<jupyter_output><empty_output><jupyter_text>If we are just concerned with viewing the summary dataframe, we can specify that dataframe to subset.<jupyter_code>just_sum_SORTED_df[just_sum_SORTED_df.comGeneName == "RPP1"]<jupyter_output><empty_output><jupyter_text>Curious about that range of cells?<jupyter_code>just_sum_SORTED_df.iloc[2531:2580]<jupyter_output><empty_output><jupyter_text>Using the `.str.contains()` method, a group of related (mostly) genes can be examined at the same time.<jupyter_code>just_sum_SORTED_df[just_sum_SORTED_df.comGeneName.str.contains("POP")]<jupyter_output><empty_output><jupyter_text>You can also easily define a list of your favorite genes to search. Here they use the SGD systematic and common names ones. If your code your subsetting method, correct you can mix and match.<jupyter_code>fav_genes = ["VPH1","YGR020C","VMA21","YPR036W","EDS1","STV1"]
just_sum_SORTED_df[(just_sum_SORTED_df.geneName.isin(fav_genes)) | (just_sum_SORTED_df.comGeneName.isin(fav_genes))]<jupyter_output><empty_output><jupyter_text>We can go beyond visualization to make reports on the ranking of genes.  In the cells below we define a gene of interest, view the pertinent row, and then use the information to make a summary about that gene.<jupyter_code>gene = "RPO21"
just_sum_SORTED_df[just_sum_SORTED_df.comGeneName == gene]
gene = "RPO21"
pos_gene = just_sum_SORTED_df[just_sum_SORTED_df.comGeneName == gene].index[0] #position in SORTED dataframe
print(f"'{gene}' ranks {pos_gene} among {len(df)} genes when sorted based on the sum of molecules/per gene for all 127 cells.")
print(f"'{gene}' ranks in the top {pos_gene/float(len(df)):.2%} of the list of genes.")<jupyter_output>'RPO21' ranks 629 among 7272 genes when sorted based on the sum of molecules/per gene for all 127 cells.
'RPO21' ranks in the top 8.65% of the list of genes.
{'fraction_non_alphanumeric': 0.09256219219598434, 'fraction_numerical': 0.02929662836216694, 'mean_word_length': 5.1923377638780295, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '193777', 'license': 'MIT', 'n_tokens_mistral': 2636, 'n_tokens_neox': 2369, 'n_words': 922, 'path': 'index.ipynb', 'repo_name': 'fomightez/BY4741_yscRNA-seq_suppl_data-binderized', 'size': '7933'}