How to use multiple spark version?

0 votes
I want to install Spark 1 and 2 without uninstalling either and use them depending on my need. How can I do this?
Dec 27, 2018 in Big Data Hadoop by digger
• 27,640 points
98 views

1 answer to this question.

0 votes

You can use the  SPARK_MAJOR_VERSION for this. Suppose you want to use version 2, set this:

export SPARK_MAJOR_VERSION=2 

Then to run, use:

spark-submit --version
answered Dec 27, 2018 by Omkar
• 67,380 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to sync Hadoop configuration files to multiple nodes?

For syncing Hadoop configuration files, you have ...READ MORE

answered Jun 21, 2018 in Big Data Hadoop by HackTheCode
162 views
0 votes
1 answer

How to find hadoop distribution and version?

Just Use the command Hadoop version ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points

edited Apr 6, 2018 by kurt_cobain 293 views
0 votes
1 answer

How to use custom FileInputFormat in MapReduce?

You have to override isSplitable method. ...READ MORE

answered Apr 10, 2018 in Big Data Hadoop by Shubham
• 13,290 points
117 views
0 votes
1 answer

How do I connect my Spark based HDInsight cluster to my blob storage?

Go through this blog: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#access-blobs I went through this ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,290 points
609 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,685 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
280 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,329 views
0 votes
1 answer

How to read Spark elements having multiple lines each?

Try this: val new_records = sc.newAPIHadoopRDD(hadoopConf,classOf[N ...READ MORE

answered Dec 12, 2018 in Big Data Hadoop by Omkar
• 67,380 points
142 views
0 votes
3 answers

Hadoop Spark: How to iterate hdfs directories?

import org.apache.hadoop.fs.{FileSystem,Path} FileSystem.get( sc.hadoopConfiguration ).listStatus( new Path("hdfs:///tmp")).foreach( ...READ MORE

answered Dec 4, 2018 in Big Data Hadoop by Komal
1,148 views