```python
from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score(clf, X, y, cv=5)
```
Where:
- `clf`: The estimator/model object to evaluate.
- `X`: The input data (features) to fit.
- `y`: The target variable/labels.
- `cv`: The number of folds for cross-validation (defaults to 5).

The function returns an array of scores (such as accuracy) for each of the $$K$$ folds. The average performance of the model can be calculated using `cv_scores.mean()`.

University of Michigan - Ann Arbor

Google

Some helpful sklearn functions include:

- KNeighborsClassifier

- DummyClassifier

- DummyRegressor

- LabelEncoder

- OneHotEncoder

- Train Test Split Function

- make_regression

- make_friedman1

- make_classification

- SVC

- DecisionTreeClassifier

- RandomForestClassifier

- RandomForestRegressor

- GridSearchCV


Helpful Scikit-Learn functions

To perform $$K$$-fold cross-validation on a dataset with $$n$$ observations, the training set is separated into $$K$$ sets by randomly sampling $$n/K$$ observations from the dataset into each fold. The model of interest is then trained on $$K-1$$ folds and validated on the left-out fold; that is, once the model is trained, we use it to make predictions on the group that was left out of training. This process repeats $$K$$ times. To evaluate the model's overall performance, we calculate the average validation error across all $$K$$ tests. $$K$$-fold cross-validation provides a robust estimate of the empirical testing error of a model and is particularly useful for selecting model designs and adjusting hyperparameters.

K-Fold Cross-Validation

In order to avoid the problem of false assumed relationships present with LabelEncoder, you can use OneHotEncoder to transform a column into a series of columns where each column represents the binary choice of 'did the original column contain value x'. This prevents the creation of false ordinal relationship. However, OneHotEncoder only takes columns of integer values greater than 1, but you can easily transform categorical variables using LabelEncoder or pd.get_dummies

sklearn.preprocessing.OneHotEncoder

Scikit-learn enables the creation of classifier objects. For example, 
```python
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors = 5, weights = 'distance')
```
In this example, the classifier object (named knn) will consider 5 neighbors in its kneighbors queries.   Setting the weights parameter to 'distance' means that closer neighbors will be more influential than those farther away  (in terms of detemrining which category the "target" point belongs to).

sklearn.neighbors.KNeighborsClassifier

Scikit-learn allows for dummy classifiers to be created for the purpose of comparing to a statistically generated model

```
dummy = DummyClassifier(strategy="most_frequent")
```

This code creates a dummy classifier which predicts every value to hold the same value as the most frequent value in the training set. 

sklearn.dummy.DummyClassifier

Scikit-learn enables allows for the transformation of categorical data into quantitative data which can be used in machine learning model training. However, an important thing to note about LabelEncoder is that it will transform your categorical data into ordinal numbers which may cause your model interpret do believe there are relationships between unrelated categories.

sklearn.preprocessing.LabelEncoder

This function randomly shuffles the dataset and splits off a certain percentage of the input samples for use as a training set, and then puts the remaining samples into a different variable for use as a test set. So in this example, we're using a 75-25% split of training versus test data. And that's a pretty standard relative split that's used. It's a good rule of thumb to use in deciding what proportion of training versus test might be helpful.
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
- Unless optional parameters test_size or train_size are passed in, the method will subset the training and test data randomly into 75% train:25% test
- random_state is an optional parameter used for reproducible output across multiple function calls

Train Test Split Function

Generates a dataset for a random regression problem. The generated dataset will be made of simulated inputs and target values based on a random linear regression model.

sklearn.datasets.make_regression

Generates the Friedman #1 regression problem. The X values will be uniformly distributed over [0,1] and the y output will be distributed according to 
```
y(X) = 10 * sin(pi * X[:,1] + 20 * (X[:,2] - 0.5) **2 + 10 * X[:,3} + 5 * X[:,4] + noise * N(0,1)
```

sklearn.datasets.make_friedman1

Generates a random classification problem. Some of the things we can specify through its parameters are the number of samples, number of classes, number of features, etc. The function returns the generated samples in an array of shape [n_samples, n_features]. It also returns an array of shape [n_samples], containing the integer labels for class membership of each sample.

sklearn.datasets.make_classification

This function creates a support vector classifier (SVC) object. Two important parameters are:
- C: Regularization parameter. In other words, this controls the cost of misclassification on the training data.
- kernel: specifies the kernel type to be used in the algorithm. This parameter can be set to "linear", "poly", "rbf", "sigmoid", or "precomputed"

sklearn.svm.SVC

Function that creates a decision tree classifier. Decision trees predict the target variable value by learning decision rules that are deduced from the data features. Some useful parameters include splitter (split strategy to use) and max_depth (maximum depth of the tree). 

sklearn.tree.DecisionTreeClassifier

Creates a random forest classifier. It will fit a number of decision tree classifiers on subsets of the data and then apply averaging over the sub-samples to improve accuracy and reduce overfitting.

 sklearn.ensemble.RandomForestClassifier

The DummyRegressor is similar to the DummyClassifier. While a DummyClassifier can serve as a baseline for comparison with real classifiers, a DummyRegressors can serve as a baseline for comparison with real regressors. 

sklearn.dummy.DummyRegressor

Performs hyperparameter tuning to determine the optimal values for the input model. Some parameters are estimator: the model we want to use for the hyperparameter tuning process, and param_grid: a dictionary in which the keys are the hyperparameters, and their corresponding values are a list of values that parameter can take.

sklearn.model_selection.GridSearchCV

Function that creates a random forest regressor. The random forest regressor uses a number of decision trees that are each trained on different subsets of the dataset. The final output is the mean of the outputs from the decision trees.

sklearn.ensemble.RandomForestRegressor

sklearn.model_selection.cross_val_score

K-Fold CV is usually the method of choice for CV, and functions as a sort of generalization of LOOCV, because:
1. It is much less computationally expensive than LOOCV.
2. It minimizes the **Bias-Variance Tradeoff**.

K-Fold Cross-Validation vs. Leave-One-Out Cross-Validation

$$5 \leq K \leq 10$$ is commonly used in Macine Learning. A too-small $K$ may lead to high bias, and a too-large $K$ may result in high variance. 

$$5 \leq K \leq 10$$ in K-Fold Cross-Validation

When evaluating numerous hyperparameter configurations, the risk of multiple testing increases, leading to situations where validation performance appears favorable simply by chance rather than reflecting true generalization capability. However, when applied to a sufficiently large dataset with a standard range of hyperparameters, $$K$$-fold cross-validation tends to be reasonably resilient against this issue, providing a more robust estimate of true error than a single validation split.

Multiple Testing Resilience in K-Fold Cross-Validation

Throughout the training and model selection process, continuously monitoring both training and validation errors is crucial for diagnosing learning behavior. If a model exhibits an extremely low training error for a specific hyperparameter configuration while simultaneously yielding a considerably higher error during $$K$$-fold cross-validation, it serves as a strong indicator of overfitting. This divergence demonstrates that the model is memorizing the training data rather than generalizing well to unseen validation folds.

Identifying Overfitting via Cross-Validation

When selecting $$K$$ for $$K$$-Fold Cross-Validation, we must consider the bias-variance tradeoff. $$K$$-Fold Cross-Validation provides $$K$$ estimates of the model error. The mean of these errors indicates the model's bias, where a lower mean value implies higher accuracy. The model variance is determined by the standard deviation of these errors, where a lower variance indicates less fluctuation in model performance. Usually, it is difficult to reduce both model bias and variance simultaneously.

Learn Before

Related