py4j protocol Py4JError org apache spark api python PythonUtils getEncryptionEnabled does not exist in the JVM

Question

I tried to integrate python with pyspark, but I am gettting this below error.

20/04/06 10:46:17 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address:
127.0.0.1; using 10.0.2.15 instead (on interface enp0s3) 20/04/06 10:46:17 WARN Utils: Set SPARK_LOCAL_IP if
you need to bind to another address 20/04/06 10:46:18 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable Using Spark's default log4j
profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging
level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Traceback (most recent call
last): File "program.py", line 5, in <module> sc = SparkContext.getOrCreate(SparkConf()) File
"/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 367, in getOrCreate SparkContext(conf=conf
or SparkConf()) File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 136, in __init__
conf, jsc, profiler_cls) File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 213, in
_do_init self._encryption_enabled = self._jvm.PythonUtils.getEncryptionEnabled(self._jsc) File
\"/root/Desktop/pyspark/spark-3.0.0-preview2-bin-hadoop2.7-1/spark-3.0.0-preview2-bin-
hadoop2.7/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1516, in __getattr__
py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM

How can I solve this error?
Thank You

Answer 1

Hi@akhtar,

This error may occur, if you don't set environment variable in .bashrc file. Set your python environment variable as follows.

export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.8.1-src.zip:$PYTHONPATH
export PATH=$SPARK_HOME/bin:$SPARK_HOME/python:$PATH

Hope this will solve your error.

If you are a beginner and need to know more about Python, It's recommended to go for Python Certification course today.

Thank You

Comments

Where can I find the bashrc file?Location?

---

You can find .bashrc file in root(/root/.bashrc).

Answer 2

Using findspark is expected to solve the problem:

Install findspark
$pip install findspark

In you code use:
import findspark
findspark.init()

Optionally you can specify "/path/to/spark" in the `init` method above;findspark.init("/path/to/spark")

Comments

I think findspark module is used to connect spark from a remote system. But this error occurs because of the python library issue.