How to assign a column in Spark Dataframe PySpark as a Primary Key

Question

I've just converted a glue dynamic frame into spark dataframe using the .todf() method. I now need to assign a column as the Primary Key. How do I do that? Please help!

Comments

What you could do is, create a dataframe on your PySpark, set the column as Primary key and then insert the values in the PySpark dataframe.

---

Hi Kalgi! I do not see a way to set a column as Primary Key in PySpark. Can you please share the details (code) about how that is done? Thanks!

Answer 1

spark do not have any concept of primary key. As spark is computation engine not database.

Comments

Yes I just read a few articles and came to the conclusion that you cannot set primary key in apache spark.