How to stop INFO messages displaying on Spark console

0 votes

I'm working on Spark console and I'd like to stop various messages that are coming on spark shell.

I tried to edit the file in order to stop these message.

Here are the contents of

# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose$exprTyper=INFO$SparkILoopInterpreter=INFO

But messages are still getting displayed on the console like the one shown below:

25/02/18 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
25/02/18 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
25/02/18 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
25/02/18 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(,44728)
25/02/18 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
25/02/18 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager with 0.0 B RAM
25/02/18 15:11:45 INFO BlockManagerMaster: Registered BlockManager
25/02/18 15:11:45 INFO HttpServer: Starting HTTP Server
25/02/18 15:11:45 INFO HttpBroadcast: Broadcast server star

Is there any way to stop these messages?

Aug 21, 2018 in Apache Spark by Meci Matt
• 9,460 points

1 answer to this question.

0 votes

Just do the following:

Edit your conf/ file and change the following line:

log4j.rootCategory=INFO, console


log4j.rootCategory=ERROR, console
It works for me, hope this will work for you as well.

answered Aug 21, 2018 by nitinrawat895
• 11,380 points

Related Questions In Apache Spark

0 votes
1 answer

How to stop messages from being displayed on spark console?

In your file you need to ...READ MORE

answered Apr 24, 2018 in Apache Spark by kurt_cobain
• 9,390 points
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

answered Mar 6, 2019 in Apache Spark by Rohit
0 votes
1 answer

How to get Spark dataset metadata?

There are a bunch of functions that ...READ MORE

answered Apr 26, 2018 in Apache Spark by kurt_cobain
• 9,390 points
0 votes
3 answers

How to transpose Spark DataFrame?

Please check the below mentioned links for ...READ MORE

answered Jan 1, 2019 in Apache Spark by anonymous
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
0 votes
1 answer

Is there any way to check the Spark version?

There are 2 ways to check the ...READ MORE

answered Apr 19, 2018 in Apache Spark by nitinrawat895
• 11,380 points
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 11,380 points
0 votes
1 answer

How to convert rdd object to dataframe in spark

SqlContext has a number of createDataFrame methods ...READ MORE

answered May 30, 2018 in Apache Spark by nitinrawat895
• 11,380 points
+1 vote
8 answers

How to replace null values in Spark DataFrame?

Hi, In Spark, fill() function of DataFrameNaFunctions class is used to replace ...READ MORE

answered Dec 15, 2020 in Apache Spark by MD
• 95,320 points