I am trying to change the default configuration of Spark Session. But it is not working.
spark_session = SparkSession.builder
.master("ip")
.enableHiveSupport()
.getOrCreate()
spark_session.conf.set("spark.executor.memory", '8g')
spark_session.conf.set('spark.executor.cores', '3')
spark_session.conf.set('spark.cores.max', '3')
spark_session.conf.set("spark.driver.memory",'8g')
sc = spark_session.sparkContext
But if I put the configuration in Spark submit, then it works fine for me.
spark-submit --master ip --executor-cores=3 --diver 8G sample.py