<jupyter_start><jupyter_text>比较下各种cv集产生的方式  KFold,stratifiedKFold,StratifiedShuffleSplit`KFold`是分成n坨，shuffle指的是拿做cv set 的那坨的顺序是随机的,若false,则按顺序抽一坨为cv set.  但不分层(即按分类的比例来构造每一坨), `StratifiedKFold`和KFold的区别是为分层，比如y中0和1为1:1,则分坨时，每坨0和1的比例也是1:1以上两种属于`不放回抽样`，即每坨的元素都是不重复的，所以的坨加起来就是原来的集合`ShuffleSplit` 和`StratifiedShuffleSplit` 则是`放回抽样`，同样分10次，每次都放回. 意味着可能10次的cv set里，元素可能重复`train_and_test_split`前面都是给出index，这个函数直接根据dataset给出分好的train set and cv set.不是Fold算法，只给出一个train和1个test, 调用多次的效果类似于`ShuffleSplit`<jupyter_code>splits = 5

tx = range(10)
ty = [0] * 5 + [1] * 5

from sklearn.model_selection import StratifiedShuffleSplit, StratifiedKFold,KFold,ShuffleSplit

from sklearn.model_selection import train_test_split
from sklearn import datasets




print("KFold shuffle=false")
kFold = KFold(n_splits=splits, shuffle=False, random_state=42)
for train_index, test_index in kFold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)
    
print("KFold shuffle=true")
kFold = KFold(n_splits=splits, shuffle=True, random_state=43)
for train_index, test_index in kFold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)

print("StratifiedKFold shuffle =false")
stratifiedKFold = StratifiedKFold(n_splits=splits, shuffle=False, random_state=42)
for train_index, test_index in stratifiedKFold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)
    
print("StratifiedKFold shuffle =true")
stratifiedKFold = StratifiedKFold(n_splits=splits, shuffle=True, random_state=42)
for train_index, test_index in stratifiedKFold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)

print("Shuffle Split")
shuffleSplit = ShuffleSplit(n_splits=splits, random_state=42, test_size=2)
for train_index, test_index in shuffleSplit.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)
    
print("Stratified Shuffle Split")
stratifiedShuffleSplit = StratifiedShuffleSplit(n_splits=splits, random_state=42, test_size=2)
for train_index, test_index in stratifiedShuffleSplit.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)
    
    
print('train_test_split:')

X_train, X_vld, y_train,y_vld = train_test_split(tx,ty,train_size=0.8,shuffle=True)
print(X_train,y_train,X_vld,y_vld)<jupyter_output>KFold shuffle=false
TRAIN: [2 3 4 5 6 7 8 9] TEST: [0 1]
TRAIN: [0 1 4 5 6 7 8 9] TEST: [2 3]
TRAIN: [0 1 2 3 6 7 8 9] TEST: [4 5]
TRAIN: [0 1 2 3 4 5 8 9] TEST: [6 7]
TRAIN: [0 1 2 3 4 5 6 7] TEST: [8 9]
KFold shuffle=true
TRAIN: [0 1 2 4 5 6 7 8] TEST: [3 9]
TRAIN: [0 1 2 3 4 5 7 9] TEST: [6 8]
TRAIN: [0 1 3 4 6 7 8 9] TEST: [2 5]
TRAIN: [0 2 3 4 5 6 8 9] TEST: [1 7]
TRAIN: [1 2 3 5 6 7 8 9] TEST: [0 4]
StratifiedKFold shuffle =false
TRAIN: [1 2 3 4 6 7 8 9] TEST: [0 5]
TRAIN: [0 2 3 4 5 7 8 9] TEST: [1 6]
TRAIN: [0 1 3 4 5 6 8 9] TEST: [2 7]
TRAIN: [0 1 2 4 5 6 7 9] TEST: [3 8]
TRAIN: [0 1 2 3 5 6 7 8] TEST: [4 9]
StratifiedKFold shuffle =true
TRAIN: [0 2 3 4 5 7 8 9] TEST: [1 6]
TRAIN: [0 1 2 3 5 6 7 8] TEST: [4 9]
TRAIN: [0 1 3 4 5 6 8 9] TEST: [2 7]
TRAIN: [1 2 3 4 6 7 8 9] TEST: [0 5]
TRAIN: [0 1 2 4 5 6 7 9] TEST: [3 8]
Shuffle Split
TRAIN: [5 0 7 2 9 4 3 6] TEST: [8 1]
TRAIN: [8 5 3 4 7 9 6 2] TEST: [0 1]
TRAIN: [0 6 8 5 3 7 1 4] TEST: [9 2]
TRAIN: [6 2 8 0 3 4 5 9] TEST: [1 7[...]
{'fraction_non_alphanumeric': 0.12767911818738517, 'fraction_numerical': 0.08450704225352113, 'mean_word_length': 5.164150943396226, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '606643', 'license': 'Apache-2.0', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1547, 'n_words': 441, 'path': 'difference_kfolds.ipynb', 'repo_name': 'anribras/machine-learning', 'size': '3266'}