University of Michigan - Ann Arbor

1. $k \ge 1$.

2. There is no structured method to find the best value for “K”. We need to try to use training data and testing data to get accuracy.

3. A too small value for k (e.g. k=1 for 100 samples) may produce results that are too noisy and will have a higher influence on the result.

4. A too large value for k may produce categories with small number of samples in each. This is referred to as the curse of dimensionality where too many dimensions for the dataset may lead to a poor prediction of a given sample. Larger values of K will have smoother decision boundaries which mean lower variance but increased bias. Also, computationally expensive.

5. Another way to choose K is though cross-validation. One way to select the cross-validation dataset from the training dataset. Take the small portion from the training dataset and call it a validation dataset, and then use the same to evaluate different possible values of K. This way we are going to predict the label for every instance in the validation set using with K equals to 1, K equals to 2, K equals to 3. and then we look at what value of K gives us the best performance on the validation set and then we can take that value and use that as the final set of our algorithm so we are minimizing the validation error.

6. In general, practice, choosing the value of k is $k = \sqrt{N}$ where N stands for the number of samples in your training dataset.

7. The Elbow method for selecting optimal k.

Specifying K = the number of the nearest neighbors for a k-Nearest Neighbors algorithm

One popular way to determine the optimal number of k is by using the elbow method. To use this method, you will run K means n times, each time adding another cluster. After each iteration you graph a score for that number of k. The point in the graph where there is an “elbow” is the point of optimal clusters. The picture below shows an example graph that would be used for determining the number of clusters -- in this example case 3 clusters is optimal


Learn Before

Related