How to merge two Spark DataFrames

0 votes

Hi Guys,

I have two data frames in Spark. I want to merge these data frames. how can I do that?

Sep 17, 2020 in Big Data Hadoop by akhtar
• 38,240 points
227 views

1 answer to this question.

0 votes

Hi@akhtar,

You can perform join operation in spark. It will merge your data frames in a single one. You can see the below example.

df1.join(df2, col(“join_key”))

If you do not want to join, but rather combine the two into a single data frame, you can use union.

df1.union(df2)
answered Sep 17, 2020 by MD
• 95,340 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to merge two consecutive regions using command line in HBase?

Hey, The Master and RegionServer both participate in ...READ MORE

answered May 29, 2019 in Big Data Hadoop by Gitika
• 65,850 points
1,341 views
0 votes
1 answer

How do I connect my Spark based HDInsight cluster to my blob storage?

Go through this blog: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#access-blobs I went through this ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,544 views
0 votes
1 answer

How to tune Spark jobs & optimize the performance?

You need to know the cluster properly ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,080 points
1,155 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
7,977 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,633 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,524 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
76,340 views
0 votes
1 answer

How to merge files in Hadoop?

Hi@akhtar, You can use getmerge command in Hadoop. ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 95,340 points
265 views
0 votes
1 answer

How to run Hadoop in Docker containers?

Hi, You can run Hadoop in Docker container. Follow ...READ MORE

answered Jan 24, 2020 in Big Data Hadoop by MD
• 95,340 points
868 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP