How to integrate Machine Learning with Spark

0 votes

Hi Guys,

I am trying to create one Machine Learning model. But it takes lots of time because of my big dataset. I want to connect a distributed system like Haddop and Spark cluster, so that it takes less time. But how can I connect ML with Spark?

May 6, 2020 in Machine Learning by akhtar
• 38,180 points
119 views

1 answer to this question.

0 votes

Hi@akhtar,

To integrate Hadoop with Spark, you need to configure a cluster first. After that, you should have the following software installed in your system.

  • Python

  • Apache Spark

  • findspark library

  • Numpy

  • Jupyter

After installing this software, open your Jupyter notebook and import findspark as follows.

$ import findspark
$ findspark.init('Replace Spark Path')

Now you are ready to do your job as you did in Machine Learning.

answered May 6, 2020 by MD
• 95,140 points

Related Questions In Machine Learning

0 votes
0 answers

How to know if a problem is solvable by machine learning?

I have recently started learning the machine. ...READ MORE

Nov 21, 2019 in Machine Learning by Hannah
• 18,550 points
209 views
0 votes
1 answer

Process for mechanical engineer to become machine learning Engineer?

Hey! There are two main languages that ...READ MORE

answered Aug 20, 2019 in Machine Learning by Vishal
326 views
+1 vote
1 answer

ValueError: could not convert string to float in Machine learning.

Hi@akhtar, You are trying to use constant variable ...READ MORE

answered Apr 14, 2020 in Machine Learning by MD
• 95,140 points
12,305 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,101 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,140 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
51,416 views
0 votes
1 answer

How to save machine learning model?

Hi@akhtar, To save your Machine Learning model, you ...READ MORE

answered Apr 13, 2020 in Machine Learning by MD
• 95,140 points
246 views
0 votes
1 answer

How to import one Machine Learning model?

Hi@akhtar, To import one pre-created ML model, you ...READ MORE

answered Apr 13, 2020 in Machine Learning by MD
• 95,140 points
144 views