<jupyter_start><jupyter_text>Data Set CurationIn data science, your model will only be as good as your data. You often don't have as much data as you'd like to have.The following are standard practices and techniques to do more with less, or at least better organize your data to ensure your modelbest reflects the real world phenomena you're attempting to model. Train, Test and Tuning<jupyter_code>import numpy as np
from sklearn.model_selection import train_test_split

# 12 data points of values 1 to 25
X = np.arange(1, 25).reshape(12, 2)
# totally made up labels
Y = np.array([0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0])

x_train, x_test, y_train, y_test = train_test_split(X, Y)

print(x_train)
print(len(x_train))
print()
print(x_test)
print(len(x_test))<jupyter_output>[[17 18]
 [13 14]
 [ 3  4]
 [15 16]
 [19 20]
 [ 7  8]
 [ 5  6]
 [11 12]
 [21 22]]
9

[[ 9 10]
 [23 24]
 [ 1  2]]
3<jupyter_text>Some options that come in handy...    - _test_size_: how much of the data is used in testing (training set because 1 - test_size)    - _random_: set a different random seed    - _stratify_: useful if you have an imbalanced data set, keeps the values of y approximately equal in proportion between test     train sets<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=4, stratify=Y)
print(x_train)
print(len(x_train))
print()
print(x_test)
print(len(x_test))<jupyter_output>[[21 22]
 [ 1  2]
 [15 16]
 [13 14]
 [17 18]
 [19 20]
 [23 24]
 [ 3  4]]
8

[[11 12]
 [ 7  8]
 [ 5  6]
 [ 9 10]]
4<jupyter_text>If you want to generate a validation set, just run train_test_split twice!This means:    - x_train = 70% of the data    - x_val = 15% of the data    - x_test = 15% of the data<jupyter_code>x_test, x_val, y_test, y_val = train_test_split(x_test, y_test, test_size=0.5, random_state=4, stratify=y_test)
print(x_train)
print(len(x_train))
print()
print(x_val)
print(len(x_val))
print()
print(x_test)
print(len(x_test))<jupyter_output>[[21 22]
 [ 1  2]
 [15 16]
 [13 14]
 [17 18]
 [19 20]
 [23 24]
 [ 3  4]]
8

[[11 12]
 [ 5  6]]
2

[[ 7  8]
 [ 9 10]]
2<jupyter_text>Cross ValidationAnother was to utilize your data set is to use Cross Validation. In this setup you use the entire labeled set, divide it intotrain and test splits K times.The KFold method will take a list and then return K lists of indices into that list representing the K test/train splits.<jupyter_code>from sklearn.model_selection import KFold
kf5 = KFold(n_splits=5, shuffle=False)

fold = 1
for train_index, test_index in kf5.split(X):
    print(f"Fold {fold}: \n\ttrain indices: {train_index} \n\ttest indices: {test_index}")
    fold +=1<jupyter_output>Fold 1: 
	train indices: [ 3  4  5  6  7  8  9 10 11] 
	test indices: [0 1 2]
Fold 2: 
	train indices: [ 0  1  2  6  7  8  9 10 11] 
	test indices: [3 4 5]
Fold 3: 
	train indices: [ 0  1  2  3  4  5  8  9 10 11] 
	test indices: [6 7]
Fold 4: 
	train indices: [ 0  1  2  3  4  5  6  7 10 11] 
	test indices: [8 9]
Fold 5: 
	train indices: [0 1 2 3 4 5 6 7 8 9] 
	test indices: [10 11]
{'fraction_non_alphanumeric': 0.10856769130150425, 'fraction_numerical': 0.07880967952910399, 'mean_word_length': 3.7573872472783827, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '162755', 'license': 'MIT', 'n_tokens_mistral': 1366, 'n_tokens_neox': 1185, 'n_words': 451, 'path': 'TrainingTips.ipynb', 'repo_name': 'nathan-gilbert/ai-ml-guild-metrics', 'size': '3058'}