<jupyter_start><jupyter_text>1 - The QCB projectThe goal of the quantitative cell biology (QCB) project is to find patterns between physical properties of our cells. Further analysis of these patterns, which may be conducted in collaboration with the modelling team, may lead to new rules that govern cell shape and potentially cell function. Two distinct approaches can be employed in this project _feature search_ and _image inspection_. 1.2 - Feature searchWe use tools from data mining to rank pairs of high-correlated non-obvious features that may lead to the discovery of new scaling laws between cell components, or clustering of cells. This approach should be conducted in collaboration with the modeling team as tools and results from both teams can be complementary.In this approach we want to use the pre-extracted features available in the database. Please, refer to this [repository](http://stash.corp.alleninstitute.org/projects/AICSAD/repos/aics-feature/browse?at=refs%2Fheads%2FfeatureCalc) for details of feature extraction. 1.3 - Image inspectionWe look at our data and we describe what we see and what features may capture that.In this approach is valid to perform simple measurements using ImageJ, for example, to test whether a particular measurement captures a particular aspect observed by eye. For instance, looking at control vs. treated images of Golgi, one may observed that treated cells display fragmented Golgi. In that case, one could use the `Image → Adjust → Auto Threshold` and `Analyze → Analyze Particles` to count the average number of fragments in each condition. If the difference in number of fragments is verified, this valuable information will guide the team responsile for feature extraction to develop Golgi-specific features that will be further available in the database. 2 - Feature extractionFeature extraction module is called aicsfeature and can be found in this [repository](http://stash.corp.alleninstitute.org/projects/AICSAD/repos/aics-feature/browse?at=refs%2Fheads%2FfeatureCalc). The module divides feature extraction in four submodules depending on the component to be analyzed, as explained in the sections below.It is important to notice that users working on the QCB project do not need to worry about the feature extraction pipeline. However, it is good to keep the pipeline in mind when presenting results and suggesting new features to be included in the database.To access the submodules for feature extraction in Python use<jupyter_code>from aicsfeature.extractor import mem, dna, fov, structure<jupyter_output><empty_output><jupyter_text>2.1 - MEM - Cell membrane submoduleThis submodule is used to extract features of cell membrane images. The features currently available are listed below.<jupyter_code>mem.GetListOfFeatures().tolist()<jupyter_output><empty_output><jupyter_text>2.2 - DNA - Nucleus submoduleThis submodule is used to extract features of nucleus images. The features currently available are listed below.<jupyter_code>dna.GetListOfFeatures().tolist()<jupyter_output><empty_output><jupyter_text>2.3 - FOV - Field of view submoduleThis submodule is used to extract features of field of view images. The features currently available are listed below.<jupyter_code>fov.GetListOfFeatures().tolist()<jupyter_output><empty_output><jupyter_text>2.4 - Structure - Structure submoduleThis submodule is used to extract features of structure images. The features currently available are listed below.<jupyter_code>structure.GetListOfFeatures().tolist()<jupyter_output>/home/matheus.viana/anaconda3/envs/qcb/lib/python3.6/site-packages/skimage/util/arraycrop.py:177: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  cropped = ar[slices]<jupyter_text>3 - Pipeline for feature extractionFeature extraction pipeline, which includes the implementation and extraction of new features for all the cells in the database, is done by Matheus and Jianxu. Pipeline1 - The file `gen_meta_ds.py` is used to first save a local version of the current meta dataset (not necessary).2 - Files `gen_dna_ds.py`, `gen_fov_ds.py`, `gen_mem_ds.py` and `gen_str_ds.py` (yet to be implemented) are used to load the meta dataset from database and apply the corresponding submodule for feature extraction.3 - The Jupyter Notebook `upload_ds.ipynb` is used to upload the new datasets of features to the database. 4 - Current available datasetsTo load the datasets of features from the database, first connect to the prod database with````import datasetdatabase as dsdbprod = dsdb.DatasetDatabase(config='/allen/aics/assay-dev/Analysis/QCB_database/prod_config.json')````Next query for the dataset you want. 4.1 QCB_MEM_featureTo load this dataset use`prod.get_dataset(name='QCB_MEM_feature')` 4.2 QCB_DNA_featureTo load this dataset use`prod.get_dataset(name='QCB_DNA_feature')` 4.3 QCB_FOV_featureTo load this dataset use`prod.get_dataset(name='QCB_FOV_feature')` 4.4 QCB_ST6GAL_featureTo load this dataset use`prod.get_dataset(name='QCB_ST6GAL_feature')` 4.5 QCB_TOM20_featureTo load this dataset use`prod.get_dataset(name='QCB_TOM20_feature')` 4.6 QCB_DRUG_MEM_featureTo load this dataset use`prod.get_dataset(name='QCB_DRUG_MEM_feature')` 4.7 QCB_DRUG_DNA_featureTo load this dataset use`prod.get_dataset(name='QCB_DRUG_DNA_feature')` 4.8 QCB_DRUG_ST6GAL_featureTo load this dataset use`prod.get_dataset(name='QCB_DRUG_ST6GAL_feature')` 4.9 QCB_DRUG_TUBA1B_featureTo load this dataset use`prod.get_dataset(name='QCB_DRUG_TUBA1B_feature')` 4.10 QCB_DRUG_SEC61B_featureTo load this dataset use`prod.get_dataset(name='QCB_DRUG_SEC61B_feature')` 5 - Checking datasets<jupyter_code>import datasetdatabase as dsdb
prod = dsdb.DatasetDatabase(config='/allen/aics/assay-dev/Analysis/QCB_database/prod_config.json')
prod.preview(name='QCB_MEM_feature')
prod.preview(name='QCB_DNA_feature')
prod.preview(name='QCB_FOV_feature')
prod.preview(name='QCB_ST6GAL_feature')
prod.preview(name='QCB_DRUG_DNA_feature')
prod.preview(name='QCB_DRUG_MEM_feature')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0688550840672538, 'fraction_numerical': 0.010568454763811049, 'mean_word_length': 6.313817330210773, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '393392', 'license': 'MIT', 'n_tokens_mistral': 1813, 'n_tokens_neox': 1731, 'n_words': 673, 'path': 'data-scripts/misc/info.ipynb', 'repo_name': 'vianamp/qcb-analysis', 'size': '6245'}