Azure HD Insight - YARN UI is not showing logs on stderr suddenly

0 votes

We used to see our application logs by going to logs-->stderr and it used to show SPARK INFO logs along with our own INFO logs, for last 1 month it is not showing anything but this:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

We are using Azure HDI for last 2+ years, since last 3-4 weeks YARN UI is not printing logs on stderr.We tried all options like using old HDI version, using our old jar file, research on log4j properties, nothing worked. We are guessing that somehow log4j output is getting redirected to somewhere else. But we never changed any config properties and dont know what to change. We also tried submitting the job using Livy as well as spark-submit, same issue.

But we should mention that if we use "println()" it works and prints on stderr.

Appreciate any thread/help please.

Mar 7, 2022 in Azure by Edureka
• 12,690 points
870 views

1 answer to this question.

0 votes

You can try the following:

In Ambari, see if the Spark History server is up and functioning.

=> Spark2 in Ambari Replace -Dlog4jspark.root.logger=INFO,RFA,Anonymizer with Dlog4jspark.root.logger=INFO,console,RFA,Anonymizer anywhere the -Dlog4jspark.root.logger=INFO,RFA,Anonymizer is used.

It can be found in the properties listed below. content,spark.driver.extraJavaOptions, spark.executor.extraJavaOptions

Whether or whether you'll be able to get anything using

  •  yarn logs -applicationId <appid> --appOwner <userid>
    

Or

   yarn logs -applicationId <appid> 
answered Mar 20, 2022 by Edureka
• 13,620 points

Related Questions In Azure

0 votes
1 answer
0 votes
0 answers

Azure Functions not showing up in Function app in portal

I've got a solution that contains a ...READ MORE

Mar 26, 2022 in Azure by Edureka
• 13,620 points
3,110 views
0 votes
1 answer
0 votes
1 answer

How to retrieve View definition on Synapse (Azure SQL DW)?

To authenticate with Azure SQL Database or ...READ MORE

answered Mar 4, 2022 in Azure by Edureka
• 13,620 points
1,698 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,992 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,508 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,594 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP