"java.lang.ClassNotFoundException" in Spark on Amazon EMR

0 votes

Hi Guys,

When I use custom JAR files in a spark-submit or PySpark job on Amazon EMR, I got this below error. 

java.lang.ClassNotFoundException error.

How can I solve this problem?

Apr 29 in Apache Spark by akhtar
• 11,270 points
63 views

1 answer to this question.

0 votes

Hi@akhtar,

In /etc/spark/conf/spark-defaults.conf, append the path of your custom JAR to the class names. 

Open vim /etc/spark/conf/spark-defaults.conf and add jars.

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

Hope this will work.

answered Apr 29 by MD
• 24,500 points

Related Questions In Apache Spark

0 votes
1 answer
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

answered Jul 11, 2019 in Apache Spark by Rajiv
1,668 views
0 votes
1 answer
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,116 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
732 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
30,400 views
0 votes
1 answer

Difference between map() and mapPartitions() function in Spark??

Hi@ akhtar, Both map() and mapPartitions() are the ...READ MORE

answered Jan 29 in Apache Spark by MD
• 24,500 points
698 views
0 votes
1 answer

Caused by: java.lang.NumberFormatException: Empty String

Hi@akhtar, As we know text files are in ...READ MORE

answered Jan 31 in Apache Spark by MD
• 24,500 points
164 views