def run(feature_sets, feature_type=cfg.default_feature_type, meta_data=None, output_dir=None, pipeline=None, train_perc=0.5, num_repetitions=200, positive_class=None, feat_sel_size=cfg.default_num_features_to_select, sub_groups='all', grid_search_level=cfg.GRIDSEARCH_LEVEL_DEFAULT, num_procs=2):
    '\n    Generate comprehensive report on the predictive performance for different feature sets and statistically compare them.\n\n    Main entry point for API access.\n\n    Parameters\n    ----------\n    feature_sets : list\n        The input can be specified in either of the following ways:\n            - list of paths to pyradigm datasets saved on disk\n            - path to a file containing list of paths (each line containing path to a valid MLDataset)\n            - list of MLDatasets that are already loaded\n            - list of tuples (to specify multiple features), each element containing (X, y) i.e. data and target labels\n            - a single tuple containing (X, y) i.e. data and target labels\n            - list of paths to CSV files, each containing one type of features.\n\n            When specifying multiple sets of input features, ensure:\n            - all of them contain the same number of samples\n            - each sample belongs to same class across all feature sets.\n\n    feature_type : str\n        String identifying the type of features as described above. It could be:\n        \'list_of_pyradigm_paths\', \'pyradigm_list\',\n        \'list_of_tuples\', \'tuple\', \'list_of_csv_paths\'\n\n    meta_data : multiple\n        The meta data can be specified in either of the following ways:\n\n            - a path to a meta data file (see :doc:`features` page)\n            - a dict keyed in by sample IDs with values representing their classes.\n            - None, if meta data is already specified in ``feature_sets`` input (e.g. with pyradigms).\n\n    pipeline : str or object\n        If a string, it identifying one of the implemented classifiers e.g. \'RandomForestClassifier\' or \'ExtraTreesClassifier\'\n        If an object, it must be a sciki-learn pipeline describing the sequence of steps.\n        This is typically a set of feature selections or dimensionality reduction steps followed by an estimator (classifier).\n\n        See http://scikit-learn.org/stable/modules/pipeline.html#pipeline for more details.\n\n        Default: None, which leads to the selection of a Random Forest classifier,\n         with robust scaling, followed by removal of low variance features.\n\n    method_names : list\n        A list of names to denote the different feature sets\n\n    out_results_dir : str\n        Path to output directory to save the cross validation results to.\n        If not specified, a new directory named \'neuropredict\' will be created in the current directory.\n\n    train_perc : float, optional\n        Percetange of subjects to train the classifier on.\n        The percentage is applied to the size of the smallest class to estimate\n        the number of subjects from each class to be reserved for training.\n        The smallest class is chosen to avoid class-imbalance in the training set.\n        Default: 0.8 (80%).\n\n    positive_class : str\n        Name of the class to be treated as positive in calculation of AUC\n\n    feat_sel_size : str or int\n        Number of features to select as part of feature selection. Options:\n\n             - \'tenth\'\n             - \'sqrt\'\n             - \'log2\'\n             - \'all\'\n\n            Default: \'tenth\' of the number of samples in the training set. For example, if your dataset has 90 samples, you chose 50 percent for training (default),  then Y will have 90*.5=45 samples in training set, leading to 5 features to be selected for taining. If you choose a fixed integer, ensure all the feature sets under evaluation have atleast that many features.\n\n    num_repetitions : int, optional\n        Number of repetitions of cross-validation estimation. Default: 200.\n\n    num_procs : int, optional\n        Number of CPUs to use to parallelize CV repetitions.\n\n        Default : 4. Number of CPUs will be capped at the number available on the machine if higher is requested.\n\n    sub_groups : list\n        This option allows the user to study different combinations of classes in a multi-class (N>2) dataset. For example, in a dataset with 3 classes CN, FTD and AD, two studies of pair-wise combinations can be studied separately with the following flag ``--sub_groups CN,FTD CN,AD``. This allows the user to focus on few interesting subgroups depending on their dataset/goal.\n\n        Format: Different subgroups must be separated by space, and each sub-group must be a comma-separated list of class names defined in the meta data file. Hence it is strongly recommended to use class names without any spaces, commas, hyphens and special characters, and ideally just alphanumeric characters separated by underscores. Any number of subgroups can be specified, but each subgroup must have atleast two distinct classes.\n\n        Default: ``\'all\'``, leading to inclusion of all available classes in a all-vs-all multi-class setting.\n\n    grid_search_level : str\n        Flag to specify the level of grid search during hyper-parameter optimization on the training set.\n        Allowed options are : \'none\', \'light\' and \'exhaustive\', in the order of how many values/values will be optimized.\n\n        More parameters and more values demand more resources and much longer time for optimization.\n\n        The \'light\' option tries to "folk wisdom" to try least number of values (no more than one or two),\n         for the parameters for the given classifier. (e.g. a lage number say 500 trees for a random forest optimization).\n         The \'light\' will be the fastest and should give a "rough idea" of predictive performance.\n         The \'exhaustive\' option will try to most parameter values for the most parameters that can be optimized.\n\n    Returns\n    -------\n    results_path : str\n        Path to pickle file containing full set of CV results.\n\n    '
    raise NotImplementedError
    return
Generate comprehensive report on the predictive performance for different feature sets and statistically compare them.

Main entry point for API access.

Parameters
----------
feature_sets : list
    The input can be specified in either of the following ways:
        - list of paths to pyradigm datasets saved on disk
        - path to a file containing list of paths (each line containing path to a valid MLDataset)
        - list of MLDatasets that are already loaded
        - list of tuples (to specify multiple features), each element containing (X, y) i.e. data and target labels
        - a single tuple containing (X, y) i.e. data and target labels
        - list of paths to CSV files, each containing one type of features.

        When specifying multiple sets of input features, ensure:
        - all of them contain the same number of samples
        - each sample belongs to same class across all feature sets.

feature_type : str
    String identifying the type of features as described above. It could be:
    'list_of_pyradigm_paths', 'pyradigm_list',
    'list_of_tuples', 'tuple', 'list_of_csv_paths'

meta_data : multiple
    The meta data can be specified in either of the following ways:

        - a path to a meta data file (see :doc:`features` page)
        - a dict keyed in by sample IDs with values representing their classes.
        - None, if meta data is already specified in ``feature_sets`` input (e.g. with pyradigms).

pipeline : str or object
    If a string, it identifying one of the implemented classifiers e.g. 'RandomForestClassifier' or 'ExtraTreesClassifier'
    If an object, it must be a sciki-learn pipeline describing the sequence of steps.
    This is typically a set of feature selections or dimensionality reduction steps followed by an estimator (classifier).

    See http://scikit-learn.org/stable/modules/pipeline.html#pipeline for more details.

    Default: None, which leads to the selection of a Random Forest classifier,
     with robust scaling, followed by removal of low variance features.

method_names : list
    A list of names to denote the different feature sets

out_results_dir : str
    Path to output directory to save the cross validation results to.
    If not specified, a new directory named 'neuropredict' will be created in the current directory.

train_perc : float, optional
    Percetange of subjects to train the classifier on.
    The percentage is applied to the size of the smallest class to estimate
    the number of subjects from each class to be reserved for training.
    The smallest class is chosen to avoid class-imbalance in the training set.
    Default: 0.8 (80%).

positive_class : str
    Name of the class to be treated as positive in calculation of AUC

feat_sel_size : str or int
    Number of features to select as part of feature selection. Options:

         - 'tenth'
         - 'sqrt'
         - 'log2'
         - 'all'

        Default: 'tenth' of the number of samples in the training set. For example, if your dataset has 90 samples, you chose 50 percent for training (default),  then Y will have 90*.5=45 samples in training set, leading to 5 features to be selected for taining. If you choose a fixed integer, ensure all the feature sets under evaluation have atleast that many features.

num_repetitions : int, optional
    Number of repetitions of cross-validation estimation. Default: 200.

num_procs : int, optional
    Number of CPUs to use to parallelize CV repetitions.

    Default : 4. Number of CPUs will be capped at the number available on the machine if higher is requested.

sub_groups : list
    This option allows the user to study different combinations of classes in a multi-class (N>2) dataset. For example, in a dataset with 3 classes CN, FTD and AD, two studies of pair-wise combinations can be studied separately with the following flag ``--sub_groups CN,FTD CN,AD``. This allows the user to focus on few interesting subgroups depending on their dataset/goal.

    Format: Different subgroups must be separated by space, and each sub-group must be a comma-separated list of class names defined in the meta data file. Hence it is strongly recommended to use class names without any spaces, commas, hyphens and special characters, and ideally just alphanumeric characters separated by underscores. Any number of subgroups can be specified, but each subgroup must have atleast two distinct classes.

    Default: ``'all'``, leading to inclusion of all available classes in a all-vs-all multi-class setting.

grid_search_level : str
    Flag to specify the level of grid search during hyper-parameter optimization on the training set.
    Allowed options are : 'none', 'light' and 'exhaustive', in the order of how many values/values will be optimized.

    More parameters and more values demand more resources and much longer time for optimization.

    The 'light' option tries to "folk wisdom" to try least number of values (no more than one or two),
     for the parameters for the given classifier. (e.g. a lage number say 500 trees for a random forest optimization).
     The 'light' will be the fastest and should give a "rough idea" of predictive performance.
     The 'exhaustive' option will try to most parameter values for the most parameters that can be optimized.

Returns
-------
results_path : str
    Path to pickle file containing full set of CV results.
def run(feature_sets, feature_type=cfg.default_feature_type, meta_data=None, output_dir=None, pipeline=None, train_perc=0.5, num_repetitions=200, positive_class=None, feat_sel_size=cfg.default_num_features_to_select, sub_groups='all', grid_search_level=cfg.GRIDSEARCH_LEVEL_DEFAULT, num_procs=2):
    '\n    Generate comprehensive report on the predictive performance for different feature sets and statistically compare them.\n\n    Main entry point for API access.\n\n    Parameters\n    ----------\n    feature_sets : list\n        The input can be specified in either of the following ways:\n            - list of paths to pyradigm datasets saved on disk\n            - path to a file containing list of paths (each line containing path to a valid MLDataset)\n            - list of MLDatasets that are already loaded\n            - list of tuples (to specify multiple features), each element containing (X, y) i.e. data and target labels\n            - a single tuple containing (X, y) i.e. data and target labels\n            - list of paths to CSV files, each containing one type of features.\n\n            When specifying multiple sets of input features, ensure:\n            - all of them contain the same number of samples\n            - each sample belongs to same class across all feature sets.\n\n    feature_type : str\n        String identifying the type of features as described above. It could be:\n        \'list_of_pyradigm_paths\', \'pyradigm_list\',\n        \'list_of_tuples\', \'tuple\', \'list_of_csv_paths\'\n\n    meta_data : multiple\n        The meta data can be specified in either of the following ways:\n\n            - a path to a meta data file (see :doc:`features` page)\n            - a dict keyed in by sample IDs with values representing their classes.\n            - None, if meta data is already specified in ``feature_sets`` input (e.g. with pyradigms).\n\n    pipeline : str or object\n        If a string, it identifying one of the implemented classifiers e.g. \'RandomForestClassifier\' or \'ExtraTreesClassifier\'\n        If an object, it must be a sciki-learn pipeline describing the sequence of steps.\n        This is typically a set of feature selections or dimensionality reduction steps followed by an estimator (classifier).\n\n        See http://scikit-learn.org/stable/modules/pipeline.html#pipeline for more details.\n\n        Default: None, which leads to the selection of a Random Forest classifier,\n         with robust scaling, followed by removal of low variance features.\n\n    method_names : list\n        A list of names to denote the different feature sets\n\n    out_results_dir : str\n        Path to output directory to save the cross validation results to.\n        If not specified, a new directory named \'neuropredict\' will be created in the current directory.\n\n    train_perc : float, optional\n        Percetange of subjects to train the classifier on.\n        The percentage is applied to the size of the smallest class to estimate\n        the number of subjects from each class to be reserved for training.\n        The smallest class is chosen to avoid class-imbalance in the training set.\n        Default: 0.8 (80%).\n\n    positive_class : str\n        Name of the class to be treated as positive in calculation of AUC\n\n    feat_sel_size : str or int\n        Number of features to select as part of feature selection. Options:\n\n             - \'tenth\'\n             - \'sqrt\'\n             - \'log2\'\n             - \'all\'\n\n            Default: \'tenth\' of the number of samples in the training set. For example, if your dataset has 90 samples, you chose 50 percent for training (default),  then Y will have 90*.5=45 samples in training set, leading to 5 features to be selected for taining. If you choose a fixed integer, ensure all the feature sets under evaluation have atleast that many features.\n\n    num_repetitions : int, optional\n        Number of repetitions of cross-validation estimation. Default: 200.\n\n    num_procs : int, optional\n        Number of CPUs to use to parallelize CV repetitions.\n\n        Default : 4. Number of CPUs will be capped at the number available on the machine if higher is requested.\n\n    sub_groups : list\n        This option allows the user to study different combinations of classes in a multi-class (N>2) dataset. For example, in a dataset with 3 classes CN, FTD and AD, two studies of pair-wise combinations can be studied separately with the following flag ``--sub_groups CN,FTD CN,AD``. This allows the user to focus on few interesting subgroups depending on their dataset/goal.\n\n        Format: Different subgroups must be separated by space, and each sub-group must be a comma-separated list of class names defined in the meta data file. Hence it is strongly recommended to use class names without any spaces, commas, hyphens and special characters, and ideally just alphanumeric characters separated by underscores. Any number of subgroups can be specified, but each subgroup must have atleast two distinct classes.\n\n        Default: ``\'all\'``, leading to inclusion of all available classes in a all-vs-all multi-class setting.\n\n    grid_search_level : str\n        Flag to specify the level of grid search during hyper-parameter optimization on the training set.\n        Allowed options are : \'none\', \'light\' and \'exhaustive\', in the order of how many values/values will be optimized.\n\n        More parameters and more values demand more resources and much longer time for optimization.\n\n        The \'light\' option tries to "folk wisdom" to try least number of values (no more than one or two),\n         for the parameters for the given classifier. (e.g. a lage number say 500 trees for a random forest optimization).\n         The \'light\' will be the fastest and should give a "rough idea" of predictive performance.\n         The \'exhaustive\' option will try to most parameter values for the most parameters that can be optimized.\n\n    Returns\n    -------\n    results_path : str\n        Path to pickle file containing full set of CV results.\n\n    '
    raise NotImplementedError
    return