Deciding number of spark context objects

–1 vote

On what basis should I decide, how many spark context objects are needed to be created while accessing a cluster?

Jan 16 in Apache Spark by digger
• 26,600 points
45 views

1 answer to this question.

0 votes

How many spark context objects you should create depends on how many jobs you want to run. Only one spark context object is created for every job so in short, number of spark jobs/application is equal to the number of spark context object or vice versa.

answered Jan 16 by Omkar
• 68,480 points

Related Questions In Apache Spark

0 votes
1 answer
0 votes
1 answer

Increase number of cores in Spark

Now that the job is already running, ...READ MORE

answered Feb 22 in Apache Spark by Reshma
208 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

answered Mar 28 in Apache Spark by Raj
246 views
+1 vote
2 answers

Hadoop 3 compatibility with older versions of Hive, Pig, Sqoop and Spark

Hadoop 3 is not widely used in ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,280 points
2,103 views
+1 vote
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,800 points
3,579 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,800 points
455 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
18,437 views
0 votes
1 answer

Spark shuffle service port number

The default port that shuffle service runs ...READ MORE

answered Mar 1 in Apache Spark by Omkar
• 68,480 points
35 views
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

answered May 3 in Apache Spark by Omkar
• 68,480 points
307 views