Linear discriminant analysis

Before we talk about linear discriminant analysis we will have a quick look on disadvantages of Logistic regression.

 Two-Class Problems. Logistic regression is intended for two-class or binary classification problems. It can be extended for multiclass classification, but is rarely used for this purpose.

 Unstable With Well Separated Classes. Logistic regression can become unstable when the classes are well separated.

 Unstable With Few Examples. Logistic regression can become unstable when there are few examples from which to estimate the parameters

 To overcome the above problems we can use Linear discriminant analysis which uses probability so we might not have the problem related to uneven classes.

But in real-time what is observed was LDA is widely used for variable reduction.

We will take the below data which we will use it to calculate the LDA.

Steps to calculate the LDA:

Step 1) Find the mean for each class  in our case we have two class (0 and 1)

Here we dont calculate the mean for the predicted variable(in our case variable Y), we need to calculate the mean for X when the class is zero and the class is one.

Mean of X when the class is zero

Mean of X when the class is one

Step 2: Find the probability of the classes.

Step 3:calculate the variance:

We already seen how to calculate variance here. We need to find variance for the X when the class is 1 and 0, same like we did for mean.

Variance of X when Y is 0 = 10.15823013

Variance of X when Y is 1 = 21.49316708

Now we need to calculate the average of variance for both the classes with degree of freedom(n-count of class)     

Step 4: Predicting using LDA.

Below is the formula for LDS

So here discriminant value of Y=0 is greater than Y=1, so we are considering Y will be 0 when x is 4.667798

Note:

For multiple variables, We need to calculate the same properties like the means and the covariance matrix(let’s say if i have x and x1 as my input variables then covariance matrix will be all combinations of x and x1 ) .These statistical properties are estimated from your data and plug into the LDA equation tomake predictions