Primary keys in apache Spark

0 votes
I have successfully established a JDBC connection with my spark and PostgreSQL. Am trying to insert some data into my database and I am using append mode but here I need to specify an id for each DataFrame.Row. Is there any other way to do it?
Jul 11 in Big Data Hadoop by nitinrawat895
• 10,110 points
21 views

1 answer to this question.

0 votes
from pyspark.sql.functions import monotonically_increasing_id
df.withColumn("id", monotonically_increasing_id()).show()

Verify the second argument of 

df.withColumn is monotonically_increasing_id() not monotonically_increasing_id.
answered Jul 11 by ravikiran
• 3,560 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to read more than one files in Apache Spark?

Try this: val text = sc.wholeTextFiles("student/*") text.collect() READ MORE

answered Dec 11, 2018 in Big Data Hadoop by Omkar
• 67,120 points
168 views
0 votes
1 answer
0 votes
1 answer

What is the Data format and database choices in Hadoop and Spark?

Use Parquet. I'm not sure about CSV ...READ MORE

answered Sep 4, 2018 in Big Data Hadoop by Frankie
• 9,810 points
48 views
0 votes
1 answer

How can I calculate exact median with Apache Spark?

You need to sort RDD and take ...READ MORE

answered Oct 8, 2018 in Big Data Hadoop by Omkar
• 67,120 points
430 views
0 votes
1 answer

What do we exactly mean by “Hadoop” – the definition of Hadoop?

The official definition of Apache Hadoop given ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by Shubham
150 views
+1 vote
1 answer
0 votes
3 answers

Can we run Spark without using Hadoop?

No, you can run spark without hadoop. ...READ MORE

answered May 7 in Big Data Hadoop by pradeep
121 views
0 votes
1 answer

Joining Multiple Spark Dataframes

You can run the below code to ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Bharani
• 4,550 points
223 views
0 votes
1 answer

Is it possible to run Apache Spark without Hadoop?

Though Spark and Hadoop were the frameworks designed ...READ MORE

answered May 2 in Big Data Hadoop by ravikiran
• 3,560 points
44 views
0 votes
1 answer

Is there a possibility to run Apache Spark without Hadoop?

Spark and Hadoop both are the open-source ...READ MORE

answered Jun 6 in Big Data Hadoop by ravikiran
• 3,560 points
24 views