Spark SQL in databricks

0 votes

How can I use if else statement in spark SQL in databricks

Eg:-

if ( drop table if exist table name )

else

create table if not exist table name(

)

Using delta

insert into table name if not exist 

else update table

Can you please help me with this?

Feb 23 in Apache Spark by Sunny
183 views

1 answer to this question.

0 votes

In sparkSql, we can use CASE when and then statement as below:

val df = spark.createDataFrame(Seq(( 2, 9), ( 1, 5),( 1, 1),( 1, 2),( 2, 8))).toDF("y", "x")

df.createOrReplaceTempView("test")

spark.sql("select CASE WHEN y = 2 THEN 'A' ELSE 'B' END AS flag, x from test").show

The output will be as below:

Spark 2.0.0

df: org.apache.spark.sql.DataFrame = [y: int, x: int]

+----+---+

|flag| x|

+----+---+

| A| 9|

| B| 5|

| B| 1|

| B| 2|

| A| 8|

+----+---+

Now, regarding your query, we can create a Hive table as below

sqlContext.sql("CREATE TABLE IF NOT EXISTS employee(id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'")

Similarly, we can drop a Hive table in the SQL context statement itself. 

answered Feb 23 by Rishi

Related Questions In Apache Spark

0 votes
1 answer

cache tables in apache spark sql

Caching the tables puts the whole table ...READ MORE

answered May 4, 2018 in Apache Spark by Data_Nerd
• 2,360 points
712 views
0 votes
1 answer

What's the difference between 'filter' and 'where' in Spark SQL?

Both 'filter' and 'where' in Spark SQL ...READ MORE

answered May 23, 2018 in Apache Spark by nitinrawat895
• 10,670 points
5,898 views
0 votes
2 answers

Sorting rows in descending order in Spark SQL

df.orderBy(org.apache.spark.sql.functions.col("columnname").desc) READ MORE

answered Jan 8 in Apache Spark by Ram Reddymasi
6,935 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

answered Mar 18 in Apache Spark by John
26 views
0 votes
1 answer
0 votes
1 answer

What do we exactly mean by “Hadoop” – the definition of Hadoop?

The official definition of Apache Hadoop given ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by Shubham
187 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,744 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
289 views
0 votes
1 answer

Not able to preserve shuffle files in Spark

You lose the files because by default, ...READ MORE

answered Feb 23 in Apache Spark by Rana
38 views
0 votes
1 answer

Why is Spark map output compressed?

Spark thinks that it is a good ...READ MORE

answered Feb 23 in Apache Spark by Wasim
51 views