In spark streaming with kafka i am getting error Exception in thread main java lang NoSuchMethodError org apache spark internal Logging init

Question

I am getting error "Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$" while running the spark streaming integration with kafka program in intellij in scala. the code is fine. I have given the versions also right. Can you please anyone tell me the problem.

I have already tried with different versions and the code is not working.

I am getting error at this place. ssc(streamingcontext):

val messages = KafkaUtils.createDirectStream[String, String](
  ssc,
  LocationStrategies.PreferConsistent,
  ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams))

Comments

What version of Scala are you using? I faced a similar problem. I was using scala 2.12 and the error was solved when I downgraded to Scale 2.12. Also check if the version is correctly mentioned in the build.sbt file.

Answer 1

Try to put the kafka client for your version.

Ex:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.11.0.3</version>
</dependency>