My own implementation of SVM and TSVM in MATLAB

Here are files of my own implementation of Support Vector Machine (SVM) & Transductive SVM (TSVM) in MATLAB.

Kernel Method

Kernels used in this project¹:

• Linear kernel: ;
• Polynomial kernel: ;
• Gaussian Kernel (RBF): ;
• Sigmoid kernel: ;

Optimization

The dual problems of SVM is a quadratic optimization problem with linear constraints. So we solved it using quadprog function of MATLAB Quadratic Programming Toolbox.

How to use it

• Step 1: Load the data. You'd better reshape the data as follow:
  • X is the matrix of input data with dimension of N-by-p where N is the number of instances and p is the number of features. For the convenience of visualization, we define p=2 here;
  • Y is the column vector of output data with dimension of N-by-1;
• Step 2: Define parameters ion define_parameters.m file.
  • poly_con is the parameter for Polynomial Kernel,
  • gamma is the parameter for Gaussian Kernel,
  • kappa1 & kappa2 are the parameters for Sigmoid Kernel,
  • precision is the tolerance of precision,
  • Cost is the hyperparameter for SVM.
• Step 3: Fit the model using SVM.m file. Choose the kernel you want and fit the model with your data.
• Step 4: Visualize the 2D plot. If the number of features of your data is 2, you can visualize your result using SVM_plot.m file.

Demo of binary classification of hard margin models

Data-set

MATLAB sample data set Fisher's 1936 iris data (fisheriris) consists of measurements on the sepal length, sepal width, petal length, and petal width for 150 iris specimens. There are 50 specimens from each of three species:

• Setosa,
• Versicolor,
• Virginica.

SVM

Support Vector Machine (SVM) [Cortes & Vapnuk, 1995] is a supervised learning model.

The following are the demo of SVM:

Transductive SVM (TSVM)

Transductive SVM (TSVM) [Joachims, 1995] is a semi-supervised learning model.

The following are the demo of TSVM:

Readings

• CMU-CS10701-Machine Learning, 2011 by Tom Mitchell: Kernel Methods, SVM
• CMU-CS10701-Machine Learning, 2011 by Tom Mitchell: SVM II
• CMU-CS10701/15781-Machine Learning, 2006 by Carlos Guestrin: Transductive SVMs
• MACHINE LEARNING WITH MISSING LABELS: TRANSDUCTIVE SVMS by Charles H Martin, PhD

Other Implementations on GitHub

• soloice/SVM-python: Implemented SVM in Python. In particular, the SMO algorithm is implemented.
• AlexanderFabisch/svm: Support Vector Machine in Python
• semisup-learn/methods/scikitTSVM.py: Semi-supervised learning frameworks for python, which allow fitting scikit-learn classifiers to partially labeled data
• CalculatedContent/tsvm: experiments testing transductive svm for my blog posts

Reference

• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3), 273-297.
• Joachims, T. (1999, June). Transductive inference for text classification using support vector machines. In ICML (Vol. 99, pp. 200-209).

Task lists

• Hard margin SVM with linear kernel
• Hard margin TSVM with linear kernel
• Add nonlinear kernels to SVM
• Add nonlinear kernels to TSVM
• Soft margin for SVM
• Soft margin for TSVM
• Add multiple level classification function
• Visualization of hyper-plane in 3D plot

---

Footnotes

1. Kernel Functions for Machine Learning Applications ↩