<jupyter_start><jupyter_text>GTEx Metadata Feature Selection- The Good: Tons and tons of metadata for GTEx!- The Bad: 301 features to go through- The Ugly: About 200 of the columns have no interpretable description and they aren't part of the description tables provided by GTEx. Google has also failed to associate GTEx with the abbreviations of the unknown headers.So let's first determine which features are actually important and worry about deriving their meaning later.<jupyter_code>gtex = pd.read_csv('gtex-sample-metadata.tsv', sep='\t')
gtex.head(2)
print 'Number of samples: {}'.format(gtex.shape[0])<jupyter_output>Number of samples: 9662<jupyter_text>Filter by the samples we know belong to our data.<jupyter_code>gtex_samples = [x.strip() for x in open('gtex-samples-in-tissue-pairs.tsv').readlines()]
gtex = gtex[gtex['SAMPID'].isin(gtex_samples)]
print 'Number of Samples: {}'.format(gtex.shape[0])
print 'Number of Features: {}'.format(gtex.shape[1])<jupyter_output>Number of Samples: 2868
Number of Features: 301<jupyter_text>Preprocessing  Apply Labels to Unlabelled Sample Attributes<jupyter_code>gtex.SMATSSCR.replace([0, 1, 2, 3], ['None', 'Mild', 'Moderate', 'Severe'], inplace=True)
gtex.DTHHRDY.replace([0, 1, 2, 3, 4], 
                     ['Ventilator', 'Violent Death', 'Fast Natural Death', 'Intermediate Death', 'Slow Death'], 
                     inplace=True)

# Drop duplicate column (Sex / Gender)
gtex.drop('GENDER', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Filter Sparse ColumnsRemove columns with too much missing data, as they are unlikely to carry much information.<jupyter_code>gtex.isnull().sum().plot(kind='hist');
n = gtex.shape[0] * 0.75  # Drop columns missing more than 25% of data

gtex.dropna(thresh=int(n), axis=1, inplace=True)
print 'Number of Features: {}'.format(gtex.shape[1])<jupyter_output>Number of Features: 272<jupyter_text>We can also drop any features that have only one unique value as they contribute no information<jupyter_code>for feature in gtex.columns:
    n = gtex[feature].unique()
    if len(n) == 1:
        print 'Dropping: {}\t{}'.format(feature, n[0])
        gtex.drop(feature, axis=1, inplace=True)
    if len(n) == 2:
        if np.nan in list(n):
            print 'Dropping: {}\t{}'.format(feature, n)
            gtex.drop(feature, axis=1, inplace=True)
print 'Number of Features: {}'.format(gtex.shape[1])<jupyter_output>Dropping: AssemblyName	GRCh37
Dropping: LibraryStrategy	RNA-Seq
Dropping: LibrarySelection	cDNA
Dropping: LibrarySource	TRANSCRIPTOMIC
Dropping: LibraryLayout	PAIRED
Dropping: Platform	ILLUMINA
Dropping: Model	Illumina HiSeq 2000
Dropping: SRAStudy	SRP012682
Dropping: BioProject	PRJNA75899
Dropping: ProjectID	75899
Dropping: SampleType	simple
Dropping: TaxID	9606
Dropping: ScientificName	Homo sapiens
Dropping: Tumor	no
Dropping: Analyte_Type	RNA:Total RNA
Dropping: CenterName	BI
Dropping: dbgap_study_accession	phs000424
Dropping: Consent	GRU
Dropping: SMCAT	['Aqua Kit: Organ and Tissue Donation' nan]
Dropping: SMMTRLTP	Tissue:PAXgene Preserved
Dropping: SMSTYP	Normal
Dropping: ANALYTE_TYPE	RNA:Total RNA
Dropping: SMAFRZE	['USE ME' nan]
Dropping: HGHTU	in
Dropping: WGHTU	lb
Dropping: DTHFUCODDU	['hour' nan]
Dropping: DTHCODDU	['hour' nan]
Dropping: DTHLUCODDU	['hour' nan]
Dropping: DTHRFGDU	['hour' nan]
Dropping: DTHVNTDU	['hour' nan]
Number of Features: 242<jupyter_text>How many of each column type do we have?<jupyter_code>gb_type = gtex.columns.to_series().groupby(gtex.dtypes).groups
gb_type = {k.name: v for k, v in gb_type.items()}
for k in gb_type:
    print 'Type: {}\tNum Features: {}'.format(k, len(gb_type[k]))<jupyter_output>Type: object	Num Features: 53
Type: int64	Num Features: 14
Type: float64	Num Features: 175<jupyter_text>All "object" types represent categorical variables (unless pandas got confused) and int64 and float64 are continuous variables.<jupyter_code>print 'Number of samples with NaNs: {} / {}'.format(gtex.loc[gtex.isnull().sum(axis=1) > 0].shape[0], gtex.shape[0])
print 'Number of columns with NaNs: {} / {}'.format(len(gtex.columns[gtex.isnull().sum() > 0]), gtex.shape[1])<jupyter_output>Number of samples with NaNs: 676 / 2868
Number of columns with NaNs: 193 / 242<jupyter_text>We have two options at this point: drop rows with missing values or attempt imputation of missing values.  1. Drop Rows with Missing Values<jupyter_code>dr = gtex.dropna()
dr_sample = dr.SAMPID
dr.shape
dr_type = dr.columns.to_series().groupby(dr.dtypes).groups
dr_type = {k.name: v for k, v in dr_type.items()}<jupyter_output><empty_output><jupyter_text>Drop categorical features with number of unique features equal to samples<jupyter_code>n = dr.shape[0]
for feature in dr[dr_type['object']]:
    if len(dr[feature].unique()) == n:
        print 'Dropping Feature: {}'.format(feature)
        dr.drop(feature, axis=1, inplace=True)
        dr_type['object'].remove(feature)
dr.shape<jupyter_output>2192 2192
Dropping Feature: Run
2192<jupyter_text>Task: Tissue Classification Drop all features related to tissue location<jupyter_code>dr_reduced = dr.drop(['Histological_Type', 'SMSMPSTE', 'SMTSD', 'SMUBRID', 'SMUBRTRM', 'SMTS', 'SMTSD', 'SMSMPSTE'], axis=1)
y = dr_reduced.Body_Site
X = dr_reduced.drop('Body_Site', axis=1)
def return_ohe_scaled_df(df):
    # Get types
    df_type = df.columns.to_series().groupby(df.dtypes).groups
    df_type = {k.name: v for k, v in df_type.items()}
    
    # One-Hot-Encode all categorical variables
    df_ohe = pd.get_dummies(df, columns=df_type['object'])
    df_ohe = df_ohe.reset_index(drop=True)
    
    # Separate out continuous variables and scale
    df_cont = scale(df[df_type['float64'] + df_type['int64']])

    robust_scaler = RobustScaler()
    df_cont_rob = robust_scaler.fit_transform(scale(df[df_type['float64'] + df_type['int64']]))
    
    # Combine
    df_X = pd.concat([df_ohe, pd.DataFrame(df_cont, columns=df_type['float64'] + df_type['int64'])], axis=1)
    df_X_rob = pd.concat([df_ohe, pd.DataFrame(df_cont_rob, columns=df_type['float64'] + df_type['int64'])], axis=1)
    return df_X, df_X_rob
X, X_rob = return_ohe_scaled_df(X)
X_train, X_test, y_train, y_test = train_test_split(X, y)
models = [RandomForestClassifier(), ExtraTreesClassifier(), GradientBoostingClassifier(), AdaBoostClassifier()]
score = 0
for model in models:
    model.fit(X_train, y_train)
    print model.score(X_test, y_test)
    if model.score(X_test, y_test) > score:
        score = model.score(X_test, y_test) 
        best_model = model
weights = pd.DataFrame()
weights['weights'] = best_model.feature_importances_
weights.index = X.columns
weights.sort_values('weights', ascending=False, inplace=True)
weights.plot()
weights.head(50)
pd.read_excel('GTEx_Data_V6_Annotations_SampleAttributesDD.xlsx')<jupyter_output><empty_output><jupyter_text>Attempt Imputation of Missing ValuesColumns with NaNs do not work well with SKLearn classifiers, so we either have to drop them or impute them.<jupyter_code>from sklearn.preprocessing import scale
from sklearn.feature_selection import VarianceThreshold
from sklearn.preprocessing import Imputer<jupyter_output><empty_output><jupyter_text>How many columns of our continuous data need to be replaced?<jupyter_code>gtex[gb_type['float64'] + gb_type['int64']].isnull().sum(axis=1) > 0<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11977186311787072, 'fraction_numerical': 0.021998913633894623, 'mean_word_length': 5.617250673854447, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '40648', 'license': 'Apache-2.0', 'n_tokens_mistral': 2759, 'n_tokens_neox': 2597, 'n_words': 677, 'path': 'Data-Collection-and-Processing/Metadata-collation/gtex-feature-selection.ipynb', 'repo_name': 'jvivian/Actionable-Subtypes-Manuscript', 'size': '7364'}