java lang ClassNotFoundException in Spark on Amazon EMR

0 votes

Hi Guys,

When I use custom JAR files in a spark-submit or PySpark job on Amazon EMR, I got this below error. 

java.lang.ClassNotFoundException error.

How can I solve this problem?

Apr 29, 2020 in Apache Spark by akhtar
• 38,170 points
899 views

1 answer to this question.

0 votes

Hi@akhtar,

In /etc/spark/conf/spark-defaults.conf, append the path of your custom JAR to the class names. 

Open vim /etc/spark/conf/spark-defaults.conf and add jars.

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

Hope this will work.

answered Apr 29, 2020 by MD
• 95,060 points

Related Questions In Apache Spark

0 votes
1 answer
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

answered Jul 11, 2019 in Apache Spark by Rajiv
3,586 views
0 votes
1 answer
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,846 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,095 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
48,271 views
+1 vote
8 answers

How to replace null values in Spark DataFrame?

Hi, In Spark, fill() function of DataFrameNaFunctions class is used to replace ...READ MORE

answered Dec 15, 2020 in Apache Spark by MD
• 95,060 points
54,249 views