How many files can hive create in HDFS duringbMapReduce job

0 votes
Is there a limit on how many files Hive can create on HDFS during a MapReduce job or can any number of files be created? I am writing a MapReduce program that involves creating files on HDFS. I wanted to know if there are any limits so that my program does not exceeds it.
May 21, 2019 in Big Data Hadoop by Sanam
543 views

1 answer to this question.

0 votes

There is a limit on how many files you can create on HDFS during MapReduce. The maximum number of files by default is set to 100000. But you can change this limit as follows:

$ hive --hiveconf hive.exec.max.created.files=<set the new limit>
answered May 21, 2019 by Charuni

Related Questions In Big Data Hadoop

+1 vote
1 answer

How can we list files in HDFS directory as per timestamp?

No, there is no other option to ...READ MORE

answered May 8, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
7,984 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 18, 2018 in Big Data Hadoop by Omkar
• 69,210 points
3,352 views
0 votes
1 answer

How to create files in hdfs?

You can create a file directly in ...READ MORE

answered Jan 25, 2019 in Big Data Hadoop by Omkar
• 69,210 points
608 views
0 votes
1 answer

How to create a Hive Database and store it in HDFS?

You will have to exclusively mention the ...READ MORE

answered May 20, 2019 in Big Data Hadoop by Jishnu
1,233 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,763 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,558 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
78,130 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
3,349 views
0 votes
1 answer

How do I create a subquery in hive?

Hey, Hive subquery is a select expression enclosed ...READ MORE

answered May 17, 2019 in Big Data Hadoop by Gitika
• 65,850 points
1,708 views
0 votes
1 answer

How can we configure remote metastore mode in Hive?

Hey,  Basically, hive-site.xml file has to be configured ...READ MORE

answered Jun 19, 2019 in Big Data Hadoop by Gitika
• 65,850 points
680 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP