How to merge two Spark DataFrames

0 votes

Hi Guys,

I have two data frames in Spark. I want to merge these data frames. how can I do that?

Sep 17, 2020 in Big Data Hadoop by akhtar
• 38,230 points
377 views

1 answer to this question.

0 votes

Hi@akhtar,

You can perform join operation in spark. It will merge your data frames in a single one. You can see the below example.

df1.join(df2, col(“join_key”))

If you do not want to join, but rather combine the two into a single data frame, you can use union.

df1.union(df2)
answered Sep 17, 2020 by MD
• 95,360 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to merge two consecutive regions using command line in HBase?

Hey, The Master and RegionServer both participate in ...READ MORE

answered May 29, 2019 in Big Data Hadoop by Gitika
• 65,890 points
1,593 views
0 votes
1 answer

How do I connect my Spark based HDInsight cluster to my blob storage?

Go through this blog: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#access-blobs I went through this ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,601 views
0 votes
1 answer

How to tune Spark jobs & optimize the performance?

You need to know the cluster properly ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,080 points
1,208 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
9,457 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
9,118 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,644 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
83,373 views
0 votes
1 answer

How to merge files in Hadoop?

Hi@akhtar, You can use getmerge command in Hadoop. ...READ MORE

answered Oct 1, 2020 in Big Data Hadoop by MD
• 95,360 points
378 views
0 votes
1 answer

How to run Hadoop in Docker containers?

Hi, You can run Hadoop in Docker container. Follow ...READ MORE

answered Jan 24, 2020 in Big Data Hadoop by MD
• 95,360 points
1,034 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP