How can I import Elastic-Net Lasso and Ridge regression in Pyspark

Question

Could you please tell me how can I use Elastic-Net, Lasso and Ridge regression in Pyspark? Actually I chose Linear, Elastic-Net, Lasso and Ridge regression these 4 algorithms according to machine learning cheatsheet. However, I don't know how to import Elastic-Net, Lasso and Ridge regression in Pyspark and cannot google the right answers. I just know using the Linear Regression in Pyspark.

Answer 1

Use the code and also refer to the documentation of apache spark.

from pyspark.ml.regression import LinearRegression
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.util import MLUtils

# Load training data
training = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt").toDF()

lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

# Fit the model
linear_Model = lr.fit(training)

# Print the weights and intercept for linear regression
print("Weights: " + str(linear_Model.weights))
print("Intercept: " + str(linear_Model.intercept))

If you look closely at the configuration, you'll notice that:

Elastic net includes both L1 and L2 regularisation as special instances when set properly. For example, a Lasso model is created by training a linear regression model with the elastic net parameter α set to 1. If α set to 0, on the other hand, the trained model is reduced to a ridge regression model.

Where:

elasticNetParam is equivalent to α and regParam is equivalent to λ.

To know more about Pyspark, it's recommended that you join Pyspark Course today.