Can we change the path where the Hive data is stored in HDFS?

0 votes
Say, in my requirement I want to keep all the hive data together, the ones which got created through Hive process and the ones which are created in Spark. Is there a way i can do it?
Jul 14 in Apache Spark by Kevin
28 views

1 answer to this question.

0 votes

Yes, you can but it has to be hdfs only. Refer to code below:

import org.apache.spark.{SparkConf, SparkContext}

import org.apache.spark.sql.{Row, SaveMode};

import org.apache.spark.sql.types.{StructType,StructField,StringType};


val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

val results = hiveContext.read.format("com.databricks.spark.avro").load("people.avro")



val schema = results.schema.map( x => x.name.concat(" ").concat( x.dataType.toString() match { case "StringType" => "STRING"} ) ).mkString(",")


val hive_sql = "CREATE EXTERNAL TABLE people_and_age (" + schema + ") ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/ravi/people_age'"

val hive_sql1 = "CREATE EXTERNAL TABLE people_and_age1 (" + schema + ") ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/ravi/people_age'"

hiveContext.sql(hive_sql)

hiveContext.sql(hive_sql1)


results.saveAsTable("people_age",SaveMode.Overwrite)

hiveContext.sql("select * from people_age").show()

hiveContext.sql("select * from people_age1").show()
answered Jul 14 by Yogi

Related Questions In Apache Spark

0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 10,670 points
5,905 views
0 votes
1 answer

Can anyone explain what is RDD in Spark?

RDD is a fundamental data structure of ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,290 points
585 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4 in Apache Spark by Dhara dhruve
864 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,745 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
289 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,602 views
0 votes
1 answer

Spark: How can i create temp views in user defined database instead of default database?

You can try the below code: df.registerTempTable(“airports”) sqlContext.sql(" create ...READ MORE

answered Jul 14 in Apache Spark by Ishan
44 views
0 votes
1 answer

Difference between cogroup and full outer join in spark

Please go through the below explanation : Full ...READ MORE

answered Jul 13 in Apache Spark by Kiran
239 views