How to stop INFO messages displaying on Spark console?

0 votes

I'm working on Spark console and I'd like to stop various messages that are coming on spark shell.

I tried to edit the file in order to stop these message.

Here are the contents of

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose$exprTyper=INFO$SparkILoopInterpreter=INFO

But messages are still getting displayed on the console like the one shown below:

25/02/18 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
25/02/18 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
25/02/18 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
25/02/18 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(,44728)
25/02/18 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
25/02/18 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager with 0.0 B RAM
25/02/18 15:11:45 INFO BlockManagerMaster: Registered BlockManager
25/02/18 15:11:45 INFO HttpServer: Starting HTTP Server
25/02/18 15:11:45 INFO HttpBroadcast: Broadcast server star

Is there any way to stop these messages?

Aug 21, 2018 in Apache Spark by Meci Matt
• 9,460 points

1 answer to this question.

0 votes

Just do the following:

Edit your conf/ file and change the following line:

log4j.rootCategory=INFO, console


log4j.rootCategory=ERROR, console
It works for me, hope this will work for you as well.

answered Aug 21, 2018 by nitinrawat895
• 10,950 points

Related Questions In Apache Spark

0 votes
1 answer

How to stop messages from being displayed on spark console?

In your file you need to ...READ MORE

answered Apr 24, 2018 in Apache Spark by kurt_cobain
• 9,320 points
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

answered Mar 6, 2019 in Apache Spark by Rohit
0 votes
1 answer

How to get Spark dataset metadata?

There are a bunch of functions that ...READ MORE

answered Apr 26, 2018 in Apache Spark by kurt_cobain
• 9,320 points
0 votes
3 answers

How to transpose Spark DataFrame?

Please check the below mentioned links for ...READ MORE

answered Dec 31, 2018 in Apache Spark by anonymous
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 5, 2018 in Big Data Hadoop by kurt_cobain
• 9,320 points
0 votes
1 answer

Is there any way to check the Spark version?

There are 2 ways to check the ...READ MORE

answered Apr 19, 2018 in Apache Spark by nitinrawat895
• 10,950 points
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 10,950 points
0 votes
1 answer

How to convert rdd object to dataframe in spark

SqlContext has a number of createDataFrame methods ...READ MORE

answered May 30, 2018 in Apache Spark by nitinrawat895
• 10,950 points
0 votes
7 answers