<jupyter_start><jupyter_text>Splitting test and training dataWhen you train a data model you may need to split up your data into test and training data setsTo accomplish this task we will use the [scikit-learn](https://scikit-learn.org/stable/) libraryscikit-learn is an open source, BSD licensed library for data science for preprocessing and training models. Before we can split our data test and training data, we need to do some data preparation<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Let's load our csv file with information about flights and flight delaysUse **shape** to find out how many rows and columns are in the original DataFrame<jupyter_code>delays_df = pd.read_csv('Data/Lots_of_flight_data.csv')
delays_df.shape<jupyter_output><empty_output><jupyter_text>Split data into features and labelsCreate a DataFrame called X containing only the features we want to use to train our model.**Note** You can only use numeric values as features, if you have non-numeric values you must apply different techniques such as Hot Encoding to convert these into numeric values before using them as features to train a model. Check out Data Science courses for more information on these techniques!<jupyter_code>X = delays_df.loc[:,['DISTANCE', 'CRS_ELAPSED_TIME']]
X.head()<jupyter_output><empty_output><jupyter_text>Create a DataFrame called y containing only the value we want to predict with our model. In our case we want to predict how many minutes late a flight will arrive. This information is in the ARR_DELAY column.<jupyter_code>y = delays_df.loc[:,['ARR_DELAY']]
y.head()<jupyter_output><empty_output><jupyter_text>Split into test and training dataUse **scikitlearn train_test_split** to move 30% of the rows into Test DataFramesThe other 70% of the rows into DataFrames we can use to train our modelNOTE: by specifying a value for *random_state* we ensure that if we run the code again the same rows will be moved into the test DataFrame. This makes our results repeatable.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
                                                    X, 
                                                    y, 
                                                    test_size=0.3, 
                                                    random_state=42
                                                   )<jupyter_output><empty_output><jupyter_text>We now have a DataFrame **X_train** which contains 70% of the rowsWe will use this DataFrame to train our model<jupyter_code>X_train.shape<jupyter_output><empty_output><jupyter_text>The DataFrame **X_test** contains the remaining 30% of the rowsWe will use this DataFrame to test our trained model, so we can check it's accuracy<jupyter_code>X_test.shape<jupyter_output><empty_output><jupyter_text>**X_train** and **X_test** contain our featuresThe features are the columns we think can help us predict how late a flight will arrive: **DISTANCE** and **CRS_ELAPSED_TIME**<jupyter_code>X_train.head()<jupyter_output><empty_output><jupyter_text>The DataFrame **y_train**  contains 70% of the rowsWe will use this DataFrame to train our model If you don't need to keep the original DataFrame, you can just delete the rows within the existing DataFrame instead of creating a new one**inplace=*True*** indicates you want to drop the rows in the specified DataFrame<jupyter_code>y_train.shape<jupyter_output><empty_output><jupyter_text>The DataFrame **y_test** contains the remaining 30% of the rowsWe will use this DataFrame to test our trained model, so we can check it's accuracy<jupyter_code>y_test.shape<jupyter_output><empty_output><jupyter_text>**y_train** and **y_test** contain our labelThe label is the columns we want to predict with our trained model: **ARR_DELAY****NOTE:**  a negative value for ARR_DELAY indicates a flight arrived early<jupyter_code>y_train.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06593131110554024, 'fraction_numerical': 0.004011030333416896, 'mean_word_length': 4.312916111850866, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '196319', 'license': 'MIT', 'n_tokens_mistral': 1125, 'n_tokens_neox': 1088, 'n_words': 449, 'path': '10 - Splitting test and training data with scikit-learn/10 - Train Test split.ipynb', 'repo_name': 'Halo-Developers/Data-Tools', 'size': '3989'}