How to merge two Spark DataFrames

0 votes

Hi Guys,

I have two data frames in Spark. I want to merge these data frames. how can I do that?

Sep 17, 2020 in Big Data Hadoop by akhtar
• 38,170 points
73 views

1 answer to this question.

0 votes

Hi@akhtar,

You can perform join operation in spark. It will merge your data frames in a single one. You can see the below example.

df1.join(df2, col(“join_key”))

If you do not want to join, but rather combine the two into a single data frame, you can use union.

df1.union(df2)
answered Sep 17, 2020 by MD
• 95,060 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to merge two consecutive regions using command line in HBase?

Hey, The Master and RegionServer both participate in ...READ MORE

answered May 29, 2019 in Big Data Hadoop by Gitika
• 65,870 points
702 views
0 votes
1 answer

How do I connect my Spark based HDInsight cluster to my blob storage?

Go through this blog: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#access-blobs I went through this ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,480 points
1,347 views
0 votes
1 answer

How to tune Spark jobs & optimize the performance?

You need to know the cluster properly ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,080 points
993 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
5,215 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,854 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,098 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
48,438 views
0 votes
1 answer

How to merge files in Hadoop?

Hi@akhtar, You can use getmerge command in Hadoop. ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 95,060 points
90 views
0 votes
1 answer

How to run Hadoop in Docker containers?

Hi, You can run Hadoop in Docker container. Follow ...READ MORE

answered Jan 24, 2020 in Big Data Hadoop by MD
• 95,060 points
550 views