java lang ClassNotFoundException in Spark on Amazon EMR

0 votes

Hi Guys,

When I use custom JAR files in a spark-submit or PySpark job on Amazon EMR, I got this below error. 

java.lang.ClassNotFoundException error.

How can I solve this problem?

Apr 29, 2020 in Apache Spark by akhtar
• 38,180 points
1,072 views

1 answer to this question.

0 votes

Hi@akhtar,

In /etc/spark/conf/spark-defaults.conf, append the path of your custom JAR to the class names. 

Open vim /etc/spark/conf/spark-defaults.conf and add jars.

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

Hope this will work.

answered Apr 29, 2020 by MD
• 95,140 points

Related Questions In Apache Spark

0 votes
1 answer
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

answered Jul 11, 2019 in Apache Spark by Rajiv
3,874 views
0 votes
1 answer
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,117 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,141 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
51,713 views
+1 vote
8 answers

How to replace null values in Spark DataFrame?

Hi, In Spark, fill() function of DataFrameNaFunctions class is used to replace ...READ MORE

answered Dec 15, 2020 in Apache Spark by MD
• 95,140 points
56,206 views