Useful Parameters for KNN Classifier

- fit(X_train, y_train)
- predict(X_test)
- score(X_test, y_test)

Useful Methods for KNN Classifier

Which of the following is true for the nearest neighbor classifier?

Scikit-learn enables the creation of classifier objects. For example, 
```python
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors = 5, weights = 'distance')
```
In this example, the classifier object (named knn) will consider 5 neighbors in its kneighbors queries.   Setting the weights parameter to 'distance' means that closer neighbors will be more influential than those farther away  (in terms of detemrining which category the "target" point belongs to).

University of Michigan - Ann Arbor

K-nearest neighbors is a non-parametric statistical method, which predicts based on the *k* nearest neighbors. k-NN classifiers are an example of what's called instance based or memory based supervised learning.

It can be applied to both classification and regression problems.
- When applied to classification problems, the prediction result would be the most common one among the *k* nearest training observations.
- When applied to regression problems, the prediction result would be the average of the *k* nearest training observations.

Some key notes about this method:
- It makes few assumptions about structure of data
- It gives potentially accurate but sometimes unstable predictions 
- It is sensitive to small changes in training data



K-nearest Neighbors (k-NN)

Some helpful sklearn functions include:

- KNeighborsClassifier

- DummyClassifier

- DummyRegressor

- LabelEncoder

- OneHotEncoder

- Train Test Split Function

- make_regression

- make_friedman1

- make_classification

- SVC

- DecisionTreeClassifier

- RandomForestClassifier

- RandomForestRegressor

- GridSearchCV


Helpful Scikit-Learn functions

https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html

Documentation for sklearn.neighbors.KNeighborsClassifier

https://scikit-learn.org/stable/user_guide.html

Scikit-learn Documentation

Statquest with Josh Starmer.(2017, June 26). Statquest: K-nearest neighbors, Clearly Explained [Video]. Youtube.  [www.youtube.com/watch?v=HVXime0nQeI](https://www.youtube.com/watch?v=HVXime0nQeI) 

Reference Video: K-Nearest Neighbors

Dhiraj, K. (2019, August 29). Difference between K-Nearest Neighbor(K-NN) and K-Means Clustering. Medium. [medium.com/@dhiraj8899/difference-between-k-nearest-neighbor-k-nn-and-k-means-clustering-d9a44859182f](https://medium.com/@dhiraj8899/difference-between-k-nearest-neighbor-k-nn-and-k-means-clustering-d9a44859182f) 

Medium: Difference between K-Means and KNN

This reference gets into mathmatical ways how KNN and K-Nearest Neighbors are different concepts

Umamaheswaran, Venkatesh. (2018, November 12). Comprehending K-means and KNN Algorithms. Medium. Retrieved From [becominghuman.ai/comprehending-k-means-and-knn-algorithms-c791be90883d](https://becominghuman.ai/comprehending-k-means-and-knn-algorithms-c791be90883d) 

Math/Python Explanation: Difference Between K-Means and KNN

A good source that talks about KNN, it's various uses, its algorithm, and examples.

Harrison, O. (2019, July 14). Machine Learning Basics with the K-Nearest Neighbors Algorithm. Retrieved March 12, 2020, from [towardsdatascience.com/machine-learning-basics-with-the-k-nearest-neighbors-algorithm-6a6e71d01761](https://towardsdatascience.com/machine-learning-basics-with-the-k-nearest-neighbors-algorithm-6a6e71d01761) 

Machine Learning Basics with KNN Algorithm

When using KNN for regression problems, the model is based off of ‘feature similarity’ which means that the output value of an observation you are trying to find is based on how closely it resembles the points in the training data set. With regression using KNN, this similarity is based on the average distance, typically using Euclidean distance, between the chosen observation and those around it.

KNN Regression

Singh, A. (2019, August 9). A Practical Introduction to K-Nearest Neighbor for Regression. Retrieved March 18, 2020, from https://www.analyticsvidhya.com/blog/2018/08/k-nearest-neighbor-introduction-regression-python/

A Practical Introduction to K-Nearest Neighbors Algorithm for Regression (Reference)

KNN can be useful in solving problems that have solutions that depend on identifying similar objects.
1. Recommender Systems
For example: Recommending products on Amazon, articles on Medium, movies on Netflix, or videos on YouTube.
2. Concept Search
This is used to search for semantically similar documents (i.e., documents containing similar topics). We could find it in many e-Discovery software packages, which are used for helping companies find all of the e-mails, contracts, etc.
3. Herta Security
They then use k-NN to identify a person by compare the face to their watchlist. 

KNN in practice

The UM online course: Applied Machine Learning in Python->Week 2-> K-Nearest Neighbors: Classification and Regression. The link is: https://www.online.umich.edu/courses/applied-machine-learning-in-python/



Reference video: K-Nearest Neighbors: Classification and Regression

sklearn.neighbors.KNeighborsClassifier

K-nearest neighbors can be used to classify data. Given a training set X_train with labels y_train, and given a new instance x_test to be classified:

- Find the k most similar instances (X_NN) to x_test that are in X_train.
- Get the labels y_NN for the instances in X_NN.
- Predict the label for x_test by combining the labels y_NN. The combination can be done through different methods, e.g., a simple majority vote can find the label that is the most common among the k most similar examples.

For example, if K = 1, then the first closest neighbor will be used to classify the unknown data point. If K = 14, then the 14 nearest neighbors will be used. If the data point falls between multiple clusters, then you choose the category that holds the most.

Classification Algorithm of K-Nearest Neighbors

Advantages
- It is easy and simple to implement
- Since it is non-parametric, you don’t have to make any assumptions about $f$
- A model does not need to be created
- It can be used for various statistical methods (classification, regression, etc.)

Disadvantages
- As the number of predictors increase, KNN can get very slow and at some point, not usable anymore.


K-Nearest Neighbors Advantages and Disadvantages

Used the data for classes 0, 1, and 2 plotted in the image.

What class would a KNeighborsClassifer classify the new point as for k = 1 and k = 3?

The $$1$$-nearest neighbor algorithm is the simplest case of the $$k$$-nearest neighbor approach, where $$k=1$$. At training time, the learner memorizes the dataset, and at prediction time, it assigns a new data point the label of its single closest neighbor based on a chosen distance function $$d$$. Because it memorizes the data, the algorithm always achieves a training error of zero by perfectly interpolating the training data. Despite this, it can still generalize; under mild conditions, the $$1$$-nearest neighbor algorithm is a consistent estimator, meaning it eventually converges to the optimal predictor.

1-Nearest Neighbor Algorithm

In nonparametric methods like the $$k$$-nearest neighbor algorithm, a distance function, denoted as $$d$$, must be specified to measure the similarity between data points. Equivalently, this involves defining a vector-valued basis function $$\phi(\mathbf{x})$$ for featurizing the data. The choice of distance metric is critical because it encodes different inductive biases and represents specific assumptions about the underlying data patterns. With a finite amount of available data, different distance functions will yield different predictors depending on how compatible these assumptions are with the observed data.

Distance Function

In order to avoid the problem of false assumed relationships present with LabelEncoder, you can use OneHotEncoder to transform a column into a series of columns where each column represents the binary choice of 'did the original column contain value x'. This prevents the creation of false ordinal relationship. However, OneHotEncoder only takes columns of integer values greater than 1, but you can easily transform categorical variables using LabelEncoder or pd.get_dummies

sklearn.preprocessing.OneHotEncoder

Scikit-learn allows for dummy classifiers to be created for the purpose of comparing to a statistically generated model

```
dummy = DummyClassifier(strategy="most_frequent")
```

This code creates a dummy classifier which predicts every value to hold the same value as the most frequent value in the training set. 

sklearn.dummy.DummyClassifier

Scikit-learn enables allows for the transformation of categorical data into quantitative data which can be used in machine learning model training. However, an important thing to note about LabelEncoder is that it will transform your categorical data into ordinal numbers which may cause your model interpret do believe there are relationships between unrelated categories.

sklearn.preprocessing.LabelEncoder

This function randomly shuffles the dataset and splits off a certain percentage of the input samples for use as a training set, and then puts the remaining samples into a different variable for use as a test set. So in this example, we're using a 75-25% split of training versus test data. And that's a pretty standard relative split that's used. It's a good rule of thumb to use in deciding what proportion of training versus test might be helpful.
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
- Unless optional parameters test_size or train_size are passed in, the method will subset the training and test data randomly into 75% train:25% test
- random_state is an optional parameter used for reproducible output across multiple function calls

Train Test Split Function

Generates a dataset for a random regression problem. The generated dataset will be made of simulated inputs and target values based on a random linear regression model.

sklearn.datasets.make_regression

Generates the Friedman #1 regression problem. The X values will be uniformly distributed over [0,1] and the y output will be distributed according to 
```
y(X) = 10 * sin(pi * X[:,1] + 20 * (X[:,2] - 0.5) **2 + 10 * X[:,3} + 5 * X[:,4] + noise * N(0,1)
```

sklearn.datasets.make_friedman1

Generates a random classification problem. Some of the things we can specify through its parameters are the number of samples, number of classes, number of features, etc. The function returns the generated samples in an array of shape [n_samples, n_features]. It also returns an array of shape [n_samples], containing the integer labels for class membership of each sample.

sklearn.datasets.make_classification

This function creates a support vector classifier (SVC) object. Two important parameters are:
- C: Regularization parameter. In other words, this controls the cost of misclassification on the training data.
- kernel: specifies the kernel type to be used in the algorithm. This parameter can be set to "linear", "poly", "rbf", "sigmoid", or "precomputed"

sklearn.svm.SVC

Function that creates a decision tree classifier. Decision trees predict the target variable value by learning decision rules that are deduced from the data features. Some useful parameters include splitter (split strategy to use) and max_depth (maximum depth of the tree). 

sklearn.tree.DecisionTreeClassifier

Creates a random forest classifier. It will fit a number of decision tree classifiers on subsets of the data and then apply averaging over the sub-samples to improve accuracy and reduce overfitting.

 sklearn.ensemble.RandomForestClassifier

The DummyRegressor is similar to the DummyClassifier. While a DummyClassifier can serve as a baseline for comparison with real classifiers, a DummyRegressors can serve as a baseline for comparison with real regressors. 

sklearn.dummy.DummyRegressor

Performs hyperparameter tuning to determine the optimal values for the input model. Some parameters are estimator: the model we want to use for the hyperparameter tuning process, and param_grid: a dictionary in which the keys are the hyperparameters, and their corresponding values are a list of values that parameter can take.

sklearn.model_selection.GridSearchCV

Function that creates a random forest regressor. The random forest regressor uses a number of decision trees that are each trained on different subsets of the dataset. The final output is the mean of the outputs from the decision trees.

sklearn.ensemble.RandomForestRegressor

from sklearn.model_selection import cross_val_score
cv_scores = cross_val_score(clf, X, y, cv)

clf is the model we want to evaluate, X is the data set, y is the corresponding ground truth target labels or values, cv is the number of folds.

By default, cross_val_score does threefold cross-validation, which will return a list of three accuracy scores, one for each of the three folds. We can take the mean value as the accuracy of the model on average.

Learn Before

Related

Learn After