How to merge two Spark DataFrames?

0 votes

Hi Guys,

I have two data frames in Spark. I want to merge these data frames. how can I do that?

Sep 17 in Big Data Hadoop by akhtar
• 28,160 points
31 views

1 answer to this question.

0 votes

Hi@akhtar,

You can perform join operation in spark. It will merge your data frames in a single one. You can see the below example.

df1.join(df2, col(“join_key”))

If you do not want to join, but rather combine the two into a single data frame, you can use union.

df1.union(df2)
answered Sep 17 by MD
• 65,200 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to merge two consecutive regions using command line in HBase?

Hey, The Master and RegionServer both participate in ...READ MORE

answered May 29, 2019 in Big Data Hadoop by Gitika
• 41,360 points
547 views
0 votes
1 answer

How do I connect my Spark based HDInsight cluster to my blob storage?

Go through this blog: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#access-blobs I went through this ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,450 points
1,191 views
0 votes
1 answer

How to tune Spark jobs & optimize the performance?

You need to know the cluster properly ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,050 points
914 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
3,962 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,950 points
6,070 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,950 points
926 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
39,008 views
0 votes
1 answer

How to merge files in Hadoop?

Hi@akhtar, You can use getmerge command in Hadoop. ...READ MORE

answered Oct 1 in Big Data Hadoop by MD
• 65,200 points
23 views
0 votes
1 answer

How to run Hadoop in Docker containers?

Hi, You can run Hadoop in Docker container. Follow ...READ MORE

answered Jan 24 in Big Data Hadoop by MD
• 65,200 points
437 views