PySpark Config ?

0 votes
What is the use of PySpark config?
Jul 26, 2018 in Apache Spark by shams
• 3,580 points
49 views

1 answer to this question.

0 votes

Mainly, we use SparkConf because we need to set a few configurations and parameters to run a Spark application on the local/cluster. In other words, SparkConf offers configurations to run a Spark application.

class pyspark.SparkConf (
  loadDefaults = True,
  _jvm = None,
  _jconf = None
)
answered Jul 26, 2018 by kurt_cobain
• 9,260 points

Related Questions In Apache Spark

0 votes
4 answers

How to change the spark Session configuration in Pyspark?

You can dynamically load properties. First create ...READ MORE

answered Dec 10, 2018 in Apache Spark by Vini
16,495 views
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 10,730 points
1,500 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

answered Jul 11 in Apache Spark by Shir
129 views
0 votes
1 answer

Pyspark is taking default path

The HDFS path for MyLab is /user/edureka_id. ...READ MORE

answered Jul 16 in Apache Spark by Khushi
33 views
0 votes
1 answer

How to call the Debug Mode in PySpark?

As far as I understand your intentions ...READ MORE

answered Jul 26 in Apache Spark by ravikiran
• 4,560 points
79 views
0 votes
1 answer

Unable to use ml library in pyspark

The error message you have shared with ...READ MORE

answered Jul 30 in Apache Spark by Karan
27 views
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 5, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
6,164 views
0 votes
1 answer

Is there any way to check the Spark version?

There are 2 ways to check the ...READ MORE

answered Apr 19, 2018 in Apache Spark by nitinrawat895
• 10,730 points
1,461 views
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 10,730 points
7,617 views