<jupyter_start><jupyter_text>In order to test if a classification score is significative a technique in repeating the classification procedure after randomizing, permuting, the labels. The p-value is then given by the percentage of runs for which the score obtained is greater than the classification score obtained in the first place.  New to Plotly?Plotly's Python library is free and open source! [Get started](https://plot.ly/python/getting-started/) by downloading the client and [reading the primer](https://plot.ly/python/getting-started/).You can set up Plotly to work in [online](https://plot.ly/python/getting-started/initialization-for-online-plotting) or [offline](https://plot.ly/python/getting-started/initialization-for-offline-plotting) mode, or in [jupyter notebooks](https://plot.ly/python/getting-started/start-plotting-online).We also have a quick-reference [cheatsheet](https://images.plot.ly/plotly-documentation/images/python_cheat_sheet.pdf) (new!) to help you get started!  Version<jupyter_code>import sklearn
sklearn.__version__<jupyter_output><empty_output><jupyter_text>Imports This tutorial imports [SVC](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.htmlsklearn.svm.SVC), [StratifiedKFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.htmlsklearn.model_selection.StratifiedKFold) and [permutation_test_score](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.permutation_test_score.htmlsklearn.model_selection.permutation_test_score).<jupyter_code>import plotly.plotly as py
import plotly.graph_objs as go

print(__doc__)

import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import permutation_test_score
from sklearn import datasets<jupyter_output>Automatically created module for IPython interactive environment<jupyter_text>Calculations Loading a dataset<jupyter_code>iris = datasets.load_iris()
X = iris.data
y = iris.target
n_classes = np.unique(y).size

# Some noisy data not correlated
random = np.random.RandomState(seed=0)
E = random.normal(size=(len(X), 2200))

# Add noisy data to the informative features for make the task harder
X = np.c_[X, E]

svm = SVC(kernel='linear')
cv = StratifiedKFold(2)

score, permutation_scores, pvalue = permutation_test_score(
    svm, X, y, scoring="accuracy", cv=cv, n_permutations=100, n_jobs=1)

print("Classification score %s (pvalue : %s)" % (score, pvalue))<jupyter_output>Classification score 0.513333333333 (pvalue : 0.00990099009901)<jupyter_text>Plot Results View histogram of permutation scores<jupyter_code>trace = go.Histogram(x=permutation_scores, 
                     nbinsx=20,
                     marker=dict(color='blue',
                                line=dict(color='black', width=1)),
                     name='Permutation scores')
trace1 = go.Scatter(x=2 * [score], 
                    y=[0, 20],
                    mode='lines',
                    line=dict(color='green', dash='dash'),
                    name='Classification Score'
                         ' (pvalue %s)' % pvalue
                   )

trace2 = go.Scatter(x=2 * [1. / n_classes], 
                    y=[1, 20], 
                    mode='lines',
                    line=dict(color='black', dash='dash'),
                    name='Luck'
                   )

data = [trace, trace1, trace2]
layout = go.Layout(xaxis=dict(title='Score'))
fig = go.Figure(data=data, layout=layout)
py.iplot(fig)<jupyter_output><empty_output><jupyter_text>License Author:              <NAME> License:             BSD 3 clause<jupyter_code>from IPython.display import display, HTML

display(HTML('<link href="//fonts.googleapis.com/css?family=Open+Sans:600,400,300,200|Inconsolata|Ubuntu+Mono:400,700" rel="stylesheet" type="text/css" />'))
display(HTML('<link rel="stylesheet" type="text/css" href="http://help.plot.ly/documentation/all_static/css/ipython-notebook-custom.css">'))

! pip install git+https://github.com/plotly/publisher.git --upgrade
import publisher
publisher.publish(
    'Test with permutations the significance of a classification score.ipynb', 'scikit-learn/plot-permutation-test-for-classification/', 'Test with Permutations the Significance of a Classification Score | plotly',
    ' ',
    title = 'Test with Permutations the Significance of a Classification Score | plotly',
    name = 'Test with Permutations the Significance of a Classification Score',
    has_thumbnail='true', thumbnail='thumbnail/j-l-bound.jpg', 
    language='scikit-learn', page_type='example_index',
    display_as='feature_selection', order=7,
    ipynb= '~Diksha_Gabha/3094')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11682539682539683, 'fraction_numerical': 0.01650793650793651, 'mean_word_length': 4.148148148148148, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '290691', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 1552, 'n_tokens_neox': 1427, 'n_words': 319, 'path': '_posts/scikit/permutations-the-significance-of-a-classification-score/Test with permutations the significance of a classification score.ipynb', 'repo_name': 'bmb804/documentation', 'size': '4737'}