Why do we use gradient descent in linear regression

0 votes
In some machine learning classes I took recently, I've covered gradient descent to find the best fit line for linear regression.

In some statistics classes, I have learnt that we can compute this line using statistic analysis, using the mean and standard deviation. Why is this seemingly more simple technique not used in machine learning?

My question is, is gradient descent the preferred method for fitting linear models? If so, why? Or did the professor simply use gradient descent in a simpler setting to introduce the class to the technique?
Mar 22, 2022 in Machine Learning by Nandini
• 5,480 points
1,386 views

1 answer to this question.

0 votes
An example you gave is one-dimensional, which is unusual in machine learning, since numerous input features are frequently present. To apply their basic approach, you must invert a matrix, which can be difficult or ill-conditioned.
The problem is usually stated as a least - square problem, which is relatively easier. Instead of using gradient descent, ordinary least square solvers could be utilized (and often are). If the number of data points is large, a typical least squares solver may be excessively expensive, but (stochastic) gradient descent could provide a solution that is as good as a more precise solution in terms of test-set error but requiring orders of magnitude less time.
If your problem is small enough to be handled quickly with an off-the-shelf least squares solver, gradient descent is probably not for you.
Because gradient descent is a general algorithm, it can be used to solve any problem involving the optimization of a cost function. The mean square error is a common cost function used in regression problems (MSE). Finding a closed form solution necessitates inverting an ill-conditioned matrix (one whose determinant is very close to zero and so does not yield a robust inverse matrix). People frequently use the gradient descent method to find a solution which does not suffer from the ill-conditionally problem to get around this problem.
answered Mar 23, 2022 by Dev
• 6,000 points

Related Questions In Machine Learning

0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

How do I create a linear regression model in Weka without training?

Weka is a classification algorithm. This is ...READ MORE

answered Mar 9, 2022 in Machine Learning by Nandini
• 5,480 points
1,344 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

What is LassoLars? - Linear regression

LassoLars is a lasso model implemented using ...READ MORE

answered May 22, 2019 in Machine Learning by Basu
2,382 views
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP