PySpark Config ?

0 votes
What is the use of PySpark config?
Jul 26, 2018 in Apache Spark by shams
• 3,580 points
33 views

1 answer to this question.

0 votes

Mainly, we use SparkConf because we need to set a few configurations and parameters to run a Spark application on the local/cluster. In other words, SparkConf offers configurations to run a Spark application.

class pyspark.SparkConf (
  loadDefaults = True,
  _jvm = None,
  _jconf = None
)
answered Jul 26, 2018 by kurt_cobain
• 9,240 points

Related Questions In Apache Spark

0 votes
4 answers

How to change the spark Session configuration in Pyspark?

You can dynamically load properties. First create ...READ MORE

answered Dec 10, 2018 in Apache Spark by Vini
10,596 views
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 10,150 points
990 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

answered Jul 11 in Apache Spark by Shir
15 views
0 votes
1 answer

Pyspark is taking default path

The HDFS path for MyLab is /user/edureka_id. ...READ MORE

answered Jul 16 in Apache Spark by Khushi
14 views
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 5, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
4,384 views
0 votes
1 answer

Is there any way to check the Spark version?

There are 2 ways to check the ...READ MORE

answered Apr 19, 2018 in Apache Spark by nitinrawat895
• 10,150 points
845 views
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 10,150 points
4,221 views
0 votes
1 answer

Using multiple config files using same template in a role in ansible

Go through this include_role module: tasks: - ...READ MORE

answered Jun 5, 2018 in Ansible by DareDev
• 6,810 points
348 views