How to integrate Machine Learning with Spark

0 votes

Hi Guys,

I am trying to create one Machine Learning model. But it takes lots of time because of my big dataset. I want to connect a distributed system like Haddop and Spark cluster, so that it takes less time. But how can I connect ML with Spark?

May 6, 2020 in Machine Learning by akhtar
• 38,230 points
536 views

1 answer to this question.

0 votes

Hi@akhtar,

To integrate Hadoop with Spark, you need to configure a cluster first. After that, you should have the following software installed in your system.

  • Python

  • Apache Spark

  • findspark library

  • Numpy

  • Jupyter

After installing this software, open your Jupyter notebook and import findspark as follows.

$ import findspark
$ findspark.init('Replace Spark Path')

Now you are ready to do your job as you did in Machine Learning.

answered May 6, 2020 by MD
• 95,440 points

Related Questions In Machine Learning

0 votes
1 answer

How to know if a problem is solvable by machine learning?

Transitioning from learning machine learning algorithms through ...READ MORE

answered Dec 13, 2023 in Machine Learning by anonymous
• 1,180 points
580 views
0 votes
1 answer
0 votes
1 answer

Process for mechanical engineer to become machine learning Engineer?

Hey! There are two main languages that ...READ MORE

answered Aug 20, 2019 in Machine Learning by Vishal

edited Oct 14, 2022 by Srinath 799 views
+1 vote
1 answer

ValueError: could not convert string to float in Machine learning.

Hi@akhtar, You are trying to use constant variable ...READ MORE

answered Apr 14, 2020 in Machine Learning by MD
• 95,440 points
25,984 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,521 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,165 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
103,809 views
0 votes
1 answer

How to save machine learning model?

Hi@akhtar, To save your Machine Learning model, you ...READ MORE

answered Apr 13, 2020 in Machine Learning by MD
• 95,440 points
686 views
0 votes
1 answer

How to import one Machine Learning model?

Hi@akhtar, To import one pre-created ML model, you ...READ MORE

answered Apr 13, 2020 in Machine Learning by MD
• 95,440 points
498 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP