PySpark Config

0 votes
What is the use of PySpark config?
Jul 26, 2018 in Apache Spark by shams
• 3,660 points
226 views

1 answer to this question.

0 votes

Mainly, we use SparkConf because we need to set a few configurations and parameters to run a Spark application on the local/cluster. In other words, SparkConf offers configurations to run a Spark application.

class pyspark.SparkConf (
  loadDefaults = True,
  _jvm = None,
  _jconf = None
)
answered Jul 26, 2018 by kurt_cobain
• 9,390 points

Related Questions In Apache Spark

0 votes
5 answers

How to change the spark Session configuration in Pyspark?

You aren't actually overwriting anything with this ...READ MORE

answered Dec 14, 2020 in Apache Spark by Gitika
• 65,950 points
64,880 views
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 11,380 points
6,404 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

answered Jul 11, 2019 in Apache Spark by Shir
1,345 views
0 votes
1 answer

Pyspark is taking default path

The HDFS path for MyLab is /user/edureka_id. ...READ MORE

answered Jul 16, 2019 in Apache Spark by Khushi
584 views
0 votes
1 answer

How to call the Debug Mode in PySpark?

As far as I understand your intentions ...READ MORE

answered Jul 26, 2019 in Apache Spark by ravikiran
• 4,620 points
2,902 views
0 votes
1 answer

Unable to use ml library in pyspark

The error message you have shared with ...READ MORE

answered Jul 30, 2019 in Apache Spark by Karan
996 views
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
13,552 views
0 votes
1 answer

Is there any way to check the Spark version?

There are 2 ways to check the ...READ MORE

answered Apr 19, 2018 in Apache Spark by nitinrawat895
• 11,380 points
5,310 views
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 11,380 points
22,288 views