Published on Dec 08,2016
10.9K Views
Email Post

We have learnt how to Build Hive and Yarn on Spark. Now let us try out Hive and Yarn examples on Spark.

Learn-Spark-Now

Hive Example on Spark

We will run an example of Hive on Spark. We will create a table, load data in that table and execute a simple query. When working with Hive, one must construct a HiveContext which inherits from SQLContext.

Command: cd spark-1.1.1

Command: ./bin/spark-shell

hive-and-yarn-practicals-on-spark-7

Create an input file ‘sample’ in your home directory as below snapshot (tab separated).

hive-and-yarn-practicals-on-spark-8

Command: val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

hive-and-yarn-practicals-on-spark-9

Command: sqlContext.sql(“CREATE TABLE IF NOT EXISTS test (name STRING, rank INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ LINES TERMINATED BY ‘\n'”)

hive-and-yarn-practicals-on-spark-10

Command: sqlContext.sql(“LOAD DATA LOCAL INPATH ‘/home/edureka/sample’ INTO TABLE test”)

hive-and-yarn-practicals-on-spark-11

Command: sqlContext.sql(“SELECT * FROM test WHERE rank < 5”).collect().foreach(println)

hive-and-yarn-practicals-on-spark-12

Yarn Example on Spark

We will run SparkPi example on Yarn. We can deploy Yarn on Spark in two modes : cluster mode and client mode. In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by Yarn on the cluster, and the client can go away after initiating the application. In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from Yarn.

Command: cd spark-1.1.1

Command: SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.1.1-hadoop2.2.0.jar ./bin/spark-submit –master yarn –deploy-mode cluster –class org.apache.spark.examples.SparkPi –num-executors 1 –driver-memory 2g –executor-memory 1g –executor-cores 1 examples/target/scala-2.10/spark-examples-1.1.1-hadoop2.2.0.jar

hive-and-yarn-practicals-on-spark-1

After you execute the above command, please wait for sometime till you get SUCCEEDED message.

hive-and-yarn-practicals-on-spark-2

Browse localhost:8088/cluster and click on the Spark application.

hive-and-yarn-practicals-on-spark-3

Click on logs.

hive-and-yarn-practicals-on-spark-4

Click on stdout to check the output.

hive-and-yarn-practicals-on-spark-5

hive-and-yarn-practicals-on-spark-6

For deploying Yarn on Spark in client mode, just make –deploy-mode as “client”. Now, you know how to build Hive and Yarn on Spark. We also did practicals on them.

Got a question for us? Please mention them in the comments section and we will get back to you.

Related Posts

Apache Spark Lighting up the Big Data World

Apache Spark with Hadoop-Why it matters?

Hive & Yarn Get Electrified By Spark

Start your Training in Apache Spark & Scala Today

About Author
Awanish
Published on Dec 08,2016
Awanish is a Sr. Research Analyst at Edureka. He has rich expertise in Big Data technologies like Hadoop, Spark, Storm, Kafka, Flink. Awanish also holds good knowledge on Devops tools like Git, Jenkins, Docker, Puppet, Nagios. He is passionate about new technologies which provide realistic solutions.

Share on

Browse Categories

Comments
2 Comments