Pyspark rdd How to get partition number in output

–1 vote
z = sc.parallelize([1,2,3,4,5,6], 2)

How to get partition number in output?

Jan 8, 2019 in Python by digger
• 26,740 points
2,535 views

1 answer to this question.

0 votes

The glom function is what you are looking for:

glom(self): Return an RDD created by coalescing all elements within each partition into a list.

a = sc.parallelize(range(10), 5)
a.glom().collect()
#output:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]

To know more about Pyspark, it's recommended that you join PySpark course today.

answered Jan 8, 2019 by Omkar
• 69,230 points

Related Questions In Python

0 votes
1 answer

How to get textual output when using exceptions in Python?

Hi, the answer is pretty simple.  Without the ...READ MORE

answered Jan 17, 2019 in Python by Nymeria
• 3,560 points
898 views
0 votes
0 answers

How to get output result from selecting two dropdown menu options in tkinter?

if i choose a value from first ...READ MORE

Sep 4, 2020 in Python by Charitra
• 120 points
3,743 views
0 votes
2 answers

How to calculate square root of a number in python?

calculate square root in python >>> import math ...READ MORE

answered Apr 2, 2019 in Python by anonymous
5,770 views
0 votes
1 answer

How to get the size of a string in Python?

If you are talking about the length ...READ MORE

answered Jun 4, 2018 in Python by aryya
• 7,460 points
1,329 views
0 votes
0 answers

try except is not working while using hdfs command

Hi,  I am trying to run following things ...READ MORE

Mar 6, 2019 in Python by anonymous
1,133 views
+1 vote
2 answers
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

answered Mar 18, 2019 in Apache Spark by John
1,185 views
0 votes
2 answers
0 votes
1 answer

How to get absolute path in python?

Instead of using os.path.dirname method which returns the relative ...READ MORE

answered Feb 4, 2019 in Python by Omkar
• 69,230 points
7,787 views
0 votes
1 answer

How to get path with filename in python?

There is no direct way to get ...READ MORE

answered Feb 4, 2019 in Python by Omkar
• 69,230 points
3,505 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP