How to integrate Machine Learning with Spark?

0 votes

Hi Guys,

I am trying to create one Machine Learning model. But it takes lots of time because of my big dataset. I want to connect a distributed system like Haddop and Spark cluster, so that it takes less time. But how can I connect ML with Spark?

May 6 in Machine Learning by akhtar
• 11,270 points
27 views

1 answer to this question.

0 votes

Hi@akhtar,

To integrate Hadoop with Spark, you need to configure a cluster first. After that you should have the these following software installed in your system.

  • Python

  • Apache Spark

  • findspark library

  • Numpy

  • Jupyter

After installing these software, open your jupyter notebook and import findspark as follows.

$ import findspark
$ findspark.init('Replace Spark Path')

Now you are ready to do your job as you did in Machine Learning.

answered May 6 by MD
• 24,500 points

Related Questions In Machine Learning

0 votes
0 answers

How to know if a problem is solvable by machine learning?

I have recently started learning the machine. ...READ MORE

Nov 21, 2019 in Machine Learning by Hannah
• 18,040 points
95 views
0 votes
1 answer

Process for mechanical engineer to become machine learning Engineer?

Hey! There are two main languages that ...READ MORE

answered Aug 20, 2019 in Machine Learning by Vishal
192 views
+1 vote
1 answer

ValueError: could not convert string to float in Machine learning.

Hi@akhtar, You are trying to use constant variable ...READ MORE

answered Apr 14 in Machine Learning by MD
• 24,500 points
1,614 views
0 votes
0 answers

What is semi-supervised machine learning?

What is semi-supervised machine learning? READ MORE

May 8, 2019 in Machine Learning by Ganesh
80 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,116 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
732 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
30,402 views
0 votes
1 answer

How to save machine learning model?

Hi@akhtar, To save your Machine Learning model, you ...READ MORE

answered Apr 13 in Machine Learning by MD
• 24,500 points
70 views
0 votes
1 answer

How to import one Machine Learning model?

Hi@akhtar, To import one pre-created ML model, you ...READ MORE

answered Apr 13 in Machine Learning by MD
• 24,500 points
44 views