How to merge two Spark DataFrames

Question

Hi Guys,

I have two data frames in Spark. I want to merge these data frames. how can I do that?

Answer 1

Hi@akhtar,

You can perform join operation in spark. It will merge your data frames in a single one. You can see the below example.

df1.join(df2, col(“join_key”))

If you do not want to join, but rather combine the two into a single data frame, you can use union.

df1.union(df2)