When there are K > 2 classes, we are able to use one-vs-one classification. The process involves selecting two pairs of classes at a time, along with a test observation that is assigned for each of the paired classes. Then, during the classification phases, we keep track of each "success" (# of times the test observation is assigned to each of the K classes). After we have the tallies of successes, the final classification is performed on the class to which the test observation had the highest # of successes out of all the paired classes.

One-versus-one in SVM

When there are K > 2 classes, we are able to use a one-vs-all classification. The process involves comparing individual K classes to the remaining K - 1 classes. It trains one classifier per class in total K classifiers. We fit K SVMs for each class to differentiate that class from all of the other classes. The final step is to assign the observation to the class which resulted in the largest parameters because we are more confident that the test observation belongs to that class compared to all of the other classes.

One-versus-all in SVM

Support Vector Machines are intended for binary classification, however there are extensions of it intended for use with more than two classes. There are two popular approaches for SVMs when we have an arbitrary number of K classes:
- One-versus-one
- One-versus-all

University of Michigan - Ann Arbor

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Support vectors are the data points that are closer to the hyperplane. These data points are important because moving them will change which location of the hyperplane maximizes the distance between points along the decision boundary, meaning that they influence the position and orientation of the hyperplane. These support vectors are used to find the maximum distance between data points of both classes (maximum margin).

It should be noted that other points in the classifier are rendered irrelevant, as only support vectors have the power to move the hyperplane.

The green linear line is the hyperplane. The distance between the black lines is the maximum margin. The data points that are circled in black are the support vectors.

What are Support Vectors?

- Hard Margin SVM (Solving the maximum margin separator problem)
- Soft Margin SVM (Allow minimally misclassified points)

Types of SVM

This video gives a general overiew of what support vector machines are and how they work.

[StatQuest with Josh Starmer]. (2019, September 30). Support Vector Machines, Clearly Explained!!! [Video File]. Retrieved from [www.youtube.com/watch?v=efR1C6CvhmE](https://www.youtube.com/watch?v=efR1C6CvhmE) 

Support Vector Machines - StatQuest Video

 SVMs with More than Two Classes

For dataset like p1.1, it's hard to use support vector classifier(soft margin classifier) since no matter where we put the classifier it will have a lot of misclassifications. 
In order to solve this problem, we can use support vector machine (SVM). The main idea behind SVM is we create a new y-axis and then square the value on x-axis. Now the value is two dimensional. We can then draw a support vector classifier that separates these two groups.

SVM is an extension of support vector classifier, which uses kernels to enlarge features in a certain way like what is depicted in the picture.

Support Vector Classifier vs Support Vector Machine

Maximal margin hyperplanes (AKA optimal separating hyperplanes) are used in order to find which hyperplane would give the greatest separation between output classes. These hyperplanes are found by computing the classifier margin.

Learn Before

Related

Learn After