How to configure Yarn from command line

0 votes

In EMR, is there a way to get a specific value of the configuration given the configuration key using the yarn command?

For example I would like to do something like this

yarn get-config yarn.scheduler.maximum-allocation-mb
Oct 16, 2018 in Big Data Hadoop by Neha
• 6,300 points
1,869 views

1 answer to this question.

0 votes

It's a bit non-intuitive, but it turns out the hdfs getconf command is capable of checking configuration properties for YARN and MapReduce, not only HDFS.

> hdfs getconf -confKey fs.defaultFS
hdfs://localhost:19000

> hdfs getconf -confKey dfs.namenode.name.dir
file:///Users/chris/hadoop-deploy-trunk/data/dfs/name

> hdfs getconf -confKey yarn.resourcemanager.address
0.0.0.0:8032

> hdfs getconf -confKey mapreduce.framework.name
yarn

A benefit of using this is that you'll see the actual, final results of any configuration properties as they are actually used by Hadoop. This would account for some of the more advanced configuration patterns, such as the use of XInclude in the XML files or property substitutions, like this:

  <property>
    <description>The address of the applications manager interface in the RM.</description>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
  </property>

Any scripting approach that tries to parse the XML files directly is unlikely to accurately match the implementation as its done inside Hadoop, so it's better to ask Hadoop itself.

You might be wondering why an hdfs command can get configuration properties for YARN and MapReduce. Great question! It's somewhat of a coincidence of the implementation needing to inject an instance of MapReduce's JobConf into some objects created via reflection. The relevant code is visible here:

https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java#L82-L114

This code is executed as part of running the hdfs getconf command. By triggering a reference to JobConf, it forces class loading and static initialization of the relevant MapReduce and YARN classes that add yarn-default.xml, yarn-site.xml, mapred-default.xml and mapred-site.xml to the set of configuration files in effect.

answered Oct 16, 2018 by Frankie
• 9,830 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to configure ulimit for Hbase using command line?

Hi, Increasing the file handles for the user ...READ MORE

answered May 29, 2019 in Big Data Hadoop by Gitika
• 65,770 points
1,041 views
0 votes
1 answer

How to see the report of Hadoop Cluster from command line?

Hi@akhtar, You can see the report of Hadoop Cluster ...READ MORE

answered Sep 25, 2020 in Big Data Hadoop by MD
• 95,460 points
3,284 views
0 votes
1 answer

How to find HDFS free space from command line?

Hi@akhtar, You can find the free space from ...READ MORE

answered Oct 5, 2020 in Big Data Hadoop by MD
• 95,460 points
850 views
0 votes
1 answer

How to start Command Line Interface in Zookeeper?

Hey, To start the CLI for zookeeper you can ...READ MORE

answered May 28, 2019 in Big Data Hadoop by Gitika
• 65,770 points
996 views
0 votes
1 answer

Apache Hadoop Yarn example program

You can go to this location $Yarn_Home/share/hadoop/mapreduce . You'll ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,221 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,059 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,562 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,991 views
0 votes
1 answer

How do I print hadoop properties in command line?

You can dump Hadoop config by running: $ ...READ MORE

answered Aug 23, 2018 in Big Data Hadoop by Frankie
• 9,830 points
2,621 views
0 votes
1 answer

How to upgrade Apache Hadoop from 2.4.1 to 2.6.0?

If the downtime is not an issue, ...READ MORE

answered Sep 7, 2018 in Big Data Hadoop by Frankie
• 9,830 points
656 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP