Regression Methods

Types of Regression Methods

1. Linear Regression
   • Models relationship between one independent and one dependent variable.
   • Best for linear relationships.
   • Uses a straight line (y = mx + c).
2. Multiple Regression
   • Models relationship between two or more independent variables and one dependent variable.
   • Still assumes a linear relationship.
3. Polynomial Regression
   • A non-linear regression method.
   • Uses a polynomial of degree n to fit data with one or more independent variables.
   • Useful when data shows curved trends.
4. Logistic Regression
   • Used to predict categorical outcomes (e.g., Yes/No, 0/1).
   • Works with binary or multi-class classification.
   • Uses a sigmoid function to output probability.
5. Lasso and Ridge Regression
   • Both are regularized linear regression techniques.
   • Help in preventing overfitting.
   • Lasso (L1) may shrink coefficients to zero, aiding in feature selection.
   • Ridge (L2) shrinks coefficients but never fully to zero.

Limitations of Regression Methods

1. Outliers
   • Extreme values can distort the regression line.
   • Model becomes biased.
2. Number of Cases
   • Need enough data: Ideal ratio is 20:1 (samples : variables).
   • Minimum 5 samples per variable in extreme cases.
3. Missing Data
   • Missing values reduce model accuracy.
   • Leads to poor fit.
4. Multicollinearity
   • Highly correlated independent variables (above 0.9) can cause bias.
   • Singularity (perfect correlation = 1) must be avoided.
   • Remedy: Remove highly correlated variables or use tolerance (1 − R²)