Azure HD Insight - YARN UI is not showing logs on stderr suddenly

Question

We used to see our application logs by going to logs-->stderr and it used to show SPARK INFO logs along with our own INFO logs, for last 1 month it is not showing anything but this:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

We are using Azure HDI for last 2+ years, since last 3-4 weeks YARN UI is not printing logs on stderr.We tried all options like using old HDI version, using our old jar file, research on log4j properties, nothing worked. We are guessing that somehow log4j output is getting redirected to somewhere else. But we never changed any config properties and dont know what to change. We also tried submitting the job using Livy as well as spark-submit, same issue.

But we should mention that if we use "println()" it works and prints on stderr.

Appreciate any thread/help please.

Answer 1

You can try the following:

In Ambari, see if the Spark History server is up and functioning.

=> Spark2 in Ambari Replace -Dlog4jspark.root.logger=INFO,RFA,Anonymizer with Dlog4jspark.root.logger=INFO,console,RFA,Anonymizer anywhere the -Dlog4jspark.root.logger=INFO,RFA,Anonymizer is used.

It can be found in the properties listed below. content,spark.driver.extraJavaOptions, spark.executor.extraJavaOptions

Whether or whether you'll be able to get anything using

•  yarn logs -applicationId <appid> --appOwner <userid>
  

Or

   yarn logs -applicationId <appid>