Write a note on support vector machines and PCA

Write a note on support vector machines and PCA

Answer:-

Support Vector Machines (SVMs):

Support Vector Machines are supervised learning algorithms primarily used for classification and regression tasks. SVMs work by finding a hyperplane that best separates data into different classes. They aim to maximize the margin, i.e., the distance between the hyperplane and the nearest data points from each class. These nearest points are called support vectors.

1. Working:
   • SVMs predict class labels based on the function f(x) = w^T x + b. If f(x) > 0, the positive class is predicted; otherwise, the negative class is predicted.
   • They use convex optimization to ensure efficient convergence.
2. Kernel Trick:
   • The kernel trick allows SVMs to work efficiently in non-linear scenarios by mapping input data into higher-dimensional spaces using kernel functions.
   • Common kernel functions:
     • Linear kernel: k(u, v) = u \cdot v
     • Gaussian (RBF) kernel: k(u, v) = \exp\left(-\frac{|u - v|^2}{2\sigma^2}\right)
   • This enables SVMs to separate data that is not linearly separable.
3. Advantages:
   • Effective in high-dimensional spaces.
   • Works well with small to medium datasets.
4. Limitations:
   • Computationally expensive for large datasets.
   • Struggles to generalize well with generic kernels on complex problems.

Principal Component Analysis (PCA):

Principal Component Analysis is an unsupervised learning algorithm used for dimensionality reduction and data representation. PCA transforms data into a lower-dimensional space while retaining as much variance as possible.

1. Working:
   • PCA learns a linear transformation z = W^T x, where W is a matrix of eigenvectors (principal components).
   • The principal components align the direction of greatest variance in the data with the axes of the new space.
   • It decorrelates the input data, ensuring that the transformed features are uncorrelated.
2. Steps:
   • Center the data by subtracting the mean.
   • Compute the covariance matrix: Var[x] = \frac{1}{m-1} X^T X.
   • Find the eigenvectors (principal components) and eigenvalues.
   • Project the data onto the new basis: z = W^T x.
3. Applications:
   • Dimensionality reduction while preserving important features.
   • Preprocessing for machine learning models to reduce noise and redundancy.
4. Advantages:
   • Simplifies data representation while maintaining maximum variance.
   • Helps visualize high-dimensional data in 2D or 3D.
5. Limitations:
   • Assumes linear relationships between features.
   • Sensitive to scaling; requires normalization of features before applying PCA.