
The motivation for using SVM’s is to generate a separating hyperplane for a binary dataset, which is **linearly separable**. 

In topological parlance, we are assuming that the positive and negative classes are contained in **disconnected** sets. 

Visually in $$\mathbb{R}^2$$, this means that we can draw a line between all of the data points, such that all of the positive points are on one side of the line, or hyperplane, and all of the negative points are on the other. 

Interestingly, a dataset might not be linearly separable in $$\mathbb{R}^k$$, but might be linearly separable in some $$d > k$$ dimensional space. More on this when talking about kernels.


Linear Decision Boundaries, and the Assumption of Hard Margin


The hard margin classifier will not even find a solution for data that is not linearly separable (in its original feature space). In that case we prefer to use a *support vector classifier*, also known as a *soft-margin* classifier. From the below figure, we see that our data is not linearly separable, due to some noisy data points. The soft margin version of SVM will allow us to build a classifier that has <100% training accuracy but will perform well with most of the observations. 


Intro to Soft-Margin SVM

- Hard Margin SVM (Solving the maximum margin separator problem)
- Soft Margin SVM (Allow minimally misclassified points)

University of Michigan - Ann Arbor

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Support vectors are the data points that are closer to the hyperplane. These data points are important because moving them will change which location of the hyperplane maximizes the distance between points along the decision boundary, meaning that they influence the position and orientation of the hyperplane. These support vectors are used to find the maximum distance between data points of both classes (maximum margin).

It should be noted that other points in the classifier are rendered irrelevant, as only support vectors have the power to move the hyperplane.

The green linear line is the hyperplane. The distance between the black lines is the maximum margin. The data points that are circled in black are the support vectors.

What are Support Vectors?

Types of SVM

This video gives a general overiew of what support vector machines are and how they work.

[StatQuest with Josh Starmer]. (2019, September 30). Support Vector Machines, Clearly Explained!!! [Video File]. Retrieved from [www.youtube.com/watch?v=efR1C6CvhmE](https://www.youtube.com/watch?v=efR1C6CvhmE) 

Support Vector Machines - StatQuest Video

Support Vector Machines are intended for binary classification, however there are extensions of it intended for use with more than two classes. There are two popular approaches for SVMs when we have an arbitrary number of K classes:
- One-versus-one
- One-versus-all

 SVMs with More than Two Classes

For dataset like p1.1, it's hard to use support vector classifier(soft margin classifier) since no matter where we put the classifier it will have a lot of misclassifications. 
In order to solve this problem, we can use support vector machine (SVM). The main idea behind SVM is we create a new y-axis and then square the value on x-axis. Now the value is two dimensional. We can then draw a support vector classifier that separates these two groups.

SVM is an extension of support vector classifier, which uses kernels to enlarge features in a certain way like what is depicted in the picture.

Support Vector Classifier vs Support Vector Machine

Maximal margin hyperplanes (AKA optimal separating hyperplanes) are used in order to find which hyperplane would give the greatest separation between output classes. These hyperplanes are found by computing the classifier margin.

Learn Before

Related

Learn After