`from sklearn.model_selection import train_test_split`

University of Michigan - Ann Arbor

This function randomly shuffles the dataset and splits off a certain percentage of the input samples for use as a training set, and then puts the remaining samples into a different variable for use as a test set. So in this example, we're using a 75-25% split of training versus test data. And that's a pretty standard relative split that's used. It's a good rule of thumb to use in deciding what proportion of training versus test might be helpful.
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
- Unless optional parameters test_size or train_size are passed in, the method will subset the training and test data randomly into 75% train:25% test
- random_state is an optional parameter used for reproducible output across multiple function calls

Train Test Split Function

What does the following code do?

```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

The key purpose of splitting the dataset into training and test sets is:

The purpose of setting the random_state parameter in train_test_split is:

Given a dataset with 10,000 observations and 50 features plus one label. Assume a train/test split of 75%/25%.

Learn Before

Related