Support vectors are the data points that are closer to the hyperplane. These data points are important because moving them will change which location of the hyperplane maximizes the distance between points along the decision boundary, meaning that they influence the position and orientation of the hyperplane. These support vectors are used to find the maximum distance between data points of both classes (maximum margin).

It should be noted that other points in the classifier are rendered irrelevant, as only support vectors have the power to move the hyperplane.

The green linear line is the hyperplane. The distance between the black lines is the maximum margin. The data points that are circled in black are the support vectors.

What are Support Vectors?

- Hard Margin SVM (Solving the maximum margin separator problem)
- Soft Margin SVM (Allow minimally misclassified points)

Types of SVM

This video gives a general overiew of what support vector machines are and how they work.

[StatQuest with Josh Starmer]. (2019, September 30). Support Vector Machines, Clearly Explained!!! [Video File]. Retrieved from [www.youtube.com/watch?v=efR1C6CvhmE](https://www.youtube.com/watch?v=efR1C6CvhmE) 

Support Vector Machines - StatQuest Video

Support Vector Machines are intended for binary classification, however there are extensions of it intended for use with more than two classes. There are two popular approaches for SVMs when we have an arbitrary number of K classes:
- One-versus-one
- One-versus-all

 SVMs with More than Two Classes

For dataset like p1.1, it's hard to use support vector classifier(soft margin classifier) since no matter where we put the classifier it will have a lot of misclassifications. 
In order to solve this problem, we can use support vector machine (SVM). The main idea behind SVM is we create a new y-axis and then square the value on x-axis. Now the value is two dimensional. We can then draw a support vector classifier that separates these two groups.

SVM is an extension of support vector classifier, which uses kernels to enlarge features in a certain way like what is depicted in the picture.

Support Vector Classifier vs Support Vector Machine

Maximal margin hyperplanes (AKA optimal separating hyperplanes) are used in order to find which hyperplane would give the greatest separation between output classes. These hyperplanes are found by computing the classifier margin.

SVM Maximal Margin Hyperplanes

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

University of Michigan - Ann Arbor

Hyperplanes can be imagined as decision boundaries that can be used to classify the data points from a dataset, which follows p - 1. Data points that are on either side can be categorized to their respective class. In addition, hyperplanes can be one or two dimensional, depending on how many input features are used. When there are two features, the hyperplane is a line (2 - 1 where p = 2) and when there are 3 then it is a flat two-dimensional plane (3 - 1 where p = 3). If there are more than 3 features (p > 3), it is very hard to conceptually visualize a hyperplane but p - 1 dimensions still apply.

What is a Hyperplane?

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

A one-dimensiona;l hyperplane is defined mathematically by the equation
$$β0 + β1X1 + β2X2 = 0$$
where each $X$ is a point $(X1,X2)$ on the hyperplane, since a 1D hyperplane is a line.

Equation of a Hyperplane

Support Vector Machines

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Artificial Neural Networks are built to resemble neurons in the human brain. The idea behind neural networks is to understand the pattern behind terms through examples rather than through task-specific rules.

Artificial Neural Networks

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

Learn Before

Related

Learn After