How many files can hive create in HDFS duringbMapReduce job?

0 votes
Is there a limit on how many files Hive can create on HDFS during a MapReduce job or can any number of files be created? I am writing a MapReduce program that involves creating files on HDFS. I wanted to know if there are any limits so that my program does not exceeds it.
May 21 in Big Data Hadoop by Sanam
47 views

1 answer to this question.

0 votes

There is a limit on how many files you can create on HDFS during MapReduce. The maximum number of files by default is set to 100000. But you can change this limit as follows:

$ hive --hiveconf hive.exec.max.created.files=<set the new limit>
answered May 21 by Charuni

Related Questions In Big Data Hadoop

0 votes
1 answer

How can we list files in HDFS directory as per timestamp?

No, there is no other option to ...READ MORE

answered May 8, 2018 in Big Data Hadoop by nitinrawat895
• 10,800 points
1,202 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 68,480 points
716 views
0 votes
1 answer

How to create files in hdfs?

You can create a file directly in ...READ MORE

answered Jan 25 in Big Data Hadoop by Omkar
• 68,480 points
122 views
0 votes
1 answer

How to create a Hive Database and store it in HDFS?

You will have to exclusively mention the ...READ MORE

answered May 20 in Big Data Hadoop by Jishnu
84 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,800 points
3,574 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,800 points
453 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
18,425 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,280 points
1,333 views
0 votes
1 answer

How do I create a subquery in hive?

Hey, Hive subquery is a select expression enclosed ...READ MORE

answered May 17 in Big Data Hadoop by Gitika
• 25,420 points
143 views
0 votes
1 answer

How can we configure remote metastore mode in Hive?

Hey,  Basically, hive-site.xml file has to be configured ...READ MORE

answered Jun 18 in Big Data Hadoop by Gitika
• 25,420 points
89 views