How to configure Yarn from command line?

0 votes

In EMR, is there a way to get a specific value of the configuration given the configuration key using the yarn command?

For example I would like to do something like this

yarn get-config yarn.scheduler.maximum-allocation-mb
Oct 16, 2018 in Big Data Hadoop by Neha
• 6,280 points
103 views

1 answer to this question.

0 votes

It's a bit non-intuitive, but it turns out the hdfs getconf command is capable of checking configuration properties for YARN and MapReduce, not only HDFS.

> hdfs getconf -confKey fs.defaultFS
hdfs://localhost:19000

> hdfs getconf -confKey dfs.namenode.name.dir
file:///Users/chris/hadoop-deploy-trunk/data/dfs/name

> hdfs getconf -confKey yarn.resourcemanager.address
0.0.0.0:8032

> hdfs getconf -confKey mapreduce.framework.name
yarn

A benefit of using this is that you'll see the actual, final results of any configuration properties as they are actually used by Hadoop. This would account for some of the more advanced configuration patterns, such as the use of XInclude in the XML files or property substitutions, like this:

  <property>
    <description>The address of the applications manager interface in the RM.</description>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
  </property>

Any scripting approach that tries to parse the XML files directly is unlikely to accurately match the implementation as its done inside Hadoop, so it's better to ask Hadoop itself.

You might be wondering why an hdfs command can get configuration properties for YARN and MapReduce. Great question! It's somewhat of a coincidence of the implementation needing to inject an instance of MapReduce's JobConf into some objects created via reflection. The relevant code is visible here:

https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java#L82-L114

This code is executed as part of running the hdfs getconf command. By triggering a reference to JobConf, it forces class loading and static initialization of the relevant MapReduce and YARN classes that add yarn-default.xml, yarn-site.xml, mapred-default.xml and mapred-site.xml to the set of configuration files in effect.

answered Oct 16, 2018 by Frankie
• 9,810 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to configure ulimit for Hbase using command line?

Hi, Increasing the file handles for the user ...READ MORE

answered May 29 in Big Data Hadoop by Gitika
• 25,340 points
28 views
0 votes
1 answer

How to start Command Line Interface in Zookeeper?

Hey, To start the CLI for zookeeper you can ...READ MORE

answered May 27 in Big Data Hadoop by Gitika
• 25,340 points
27 views
0 votes
1 answer

How to merge two consecutive regions using command line in HBase?

Hey, The Master and RegionServer both participate in ...READ MORE

answered May 29 in Big Data Hadoop by Gitika
• 25,340 points
35 views
0 votes
1 answer

How to create the Oozie DB using the 'ooziedb.sh' command line tool?

This is the command you can use: $ ...READ MORE

answered Jun 10 in Big Data Hadoop by Gitika
• 25,340 points
37 views
0 votes
1 answer

Apache Hadoop Yarn example program

You can go to this location $Yarn_Home/share/hadoop/mapreduce . You'll ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by nitinrawat895
• 10,670 points
222 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,648 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
275 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,206 views
0 votes
1 answer

How do I print hadoop properties in command line?

You can dump Hadoop config by running: $ ...READ MORE

answered Aug 23, 2018 in Big Data Hadoop by Frankie
• 9,810 points
136 views
0 votes
1 answer

How to upgrade Apache Hadoop from 2.4.1 to 2.6.0?

If the downtime is not an issue, ...READ MORE

answered Sep 7, 2018 in Big Data Hadoop by Frankie
• 9,810 points
27 views