How to stop messages from being displayed on spark console?

0 votes

I'm receiving various log msgs on spark shell and I want to remove them. On a website, I came to know that it can be done by editing the log4j.properties file

Here are the contents of log4j.properties

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
Apr 24, 2018 in Apache Spark by Ashish
• 2,630 points
1,049 views

1 answer to this question.

0 votes

In your log4j.properties file you need to replace this line 

log4j.rootCategory=INFO, console

with

log4j.rootCategory=ERROR, console.




Hope it helps
answered Apr 24, 2018 by kurt_cobain
• 9,240 points

Related Questions In Apache Spark

0 votes
1 answer

How to stop INFO messages displaying on Spark console?

Just do the following: Edit your conf/log4j.properties file ...READ MORE

answered Aug 21, 2018 in Apache Spark by nitinrawat895
• 10,670 points
338 views
0 votes
1 answer

How to save and retrieve the Spark RDD from HDFS?

You can save the RDD using saveAsObjectFile and saveAsTextFile method. ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,290 points
1,954 views
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

answered Mar 6 in Apache Spark by Rohit
35 views
0 votes
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6 in Apache Spark by Gitika
• 25,340 points
86 views
0 votes
1 answer
+1 vote
2 answers

Execute Pig Script from Grunt Shell

From your current directory run  pig -x local Then ...READ MORE

answered Oct 25, 2018 in Big Data Hadoop by Kunal
745 views
0 votes
1 answer
0 votes
2 answers

ansible-command not found

Use some other variable instead of PATH. READ MORE

answered Apr 23 in Ansible by Vismaya
2,813 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,240 points
1,209 views
0 votes
1 answer

How to get Spark dataset metadata?

There are a bunch of functions that ...READ MORE

answered Apr 26, 2018 in Apache Spark by kurt_cobain
• 9,240 points
359 views