Is it better to have one large parquet file or lots of smaller parquet files?

0 votes
I know that hdfs will split files into 64mb chunks. We have streaming data coming in and we can store them to large files or medium sized files.
So my question is that What is the optimum size for columnar file storage?
Would smaller files save any computation time over having, say, 1gb files?
May 23, 2018 in Apache Spark by Shubham
• 13,300 points
2,157 views

1 answer to this question.

0 votes

Ideally, you would use snappy compression (default) due to snappy compressed parquet files being splittable. 

Using snappy instead of gzip will significantly increase the file size, so if storage space is an issue, that needs to be considered.

.option("compression", "gzip") is the option to override the default snappy compression.

If you need to resize/repartition your Dataset/DataFrame/RDD, call the .coalesce(<num_partitions> or worst case .repartition(<num_partitions>) function.

Also, parquet file size and for that matter all files generally should be greater in size than the HDFS block size (default 128MB).

Refer the following links to know more: 

https://forums.databricks.com/questions/101/what-is-an-optimal-size-for-file-partitions-using.html 

http://boristyukin.com/is-snappy-compressed-parquet-file-splittable/

Hope this will help you!

answered May 23, 2018 by nitinrawat895
• 10,670 points

Related Questions In Apache Spark

0 votes
1 answer

How to use yield keyword in scala and why it is used instead of println?

Hi, The yield keyword is used because the ...READ MORE

answered Jul 5 in Apache Spark by Gitika
• 25,340 points
66 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,240 points
119 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,240 points
1,305 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
1,094 views
+1 vote
1 answer
0 votes
1 answer

“no such file or directory" in case of hadoop fs -ls

The behaviour that you are seeing is ...READ MORE

answered May 9, 2018 in Big Data Hadoop by nitinrawat895
• 10,670 points

edited May 9, 2018 by nitinrawat895 2,394 views
0 votes
1 answer
0 votes
1 answer

Which is better in term of speed, Shark or Spark?

Spark is a framework for distributed data ...READ MORE

answered Jun 25, 2018 in Apache Spark by nitinrawat895
• 10,670 points
36 views
0 votes
1 answer

Is it mandatory to start Hadoop to run spark application?

No, it is not mandatory, but there ...READ MORE

answered Jun 14, 2018 in Apache Spark by nitinrawat895
• 10,670 points
56 views