How to add third party java jars for use in PySpark

Question

For my project, I required some third party Database client libraries in Java. I want to access them through

java_gateway.py
E.g: to make the client class (not a jdbc driver!) available to the python client via the java gateway:

java_import(gateway.jvm, "org.mydatabase.MyDBClient")
It is not clear where to add the third party libraries to the jvm classpath. I tried to add to compute-classpath.sh but that did nto seem to work: I get

 Py4jError: Trying to call a package
Also, when comparing to Hive: the hive jar files are NOT loaded via compute-classpath.sh so that makes me suspicious. There seems to be some other mechanism happening to set up the jvm side classpath.

Can someone help?

Thanks in advance!

Answer 1

You can add external jars as arguments to PySpark.

pyspark --jars file1.jar,file2.jar

Hope this helps!

To know more about Pyspark, it's recommended that you join Pyspark course online.

Thanks.