How many files can hive create in HDFS duringbMapReduce job?

0 votes
Is there a limit on how many files Hive can create on HDFS during a MapReduce job or can any number of files be created? I am writing a MapReduce program that involves creating files on HDFS. I wanted to know if there are any limits so that my program does not exceeds it.
May 21, 2019 in Big Data Hadoop by Sanam
124 views

1 answer to this question.

0 votes

There is a limit on how many files you can create on HDFS during MapReduce. The maximum number of files by default is set to 100000. But you can change this limit as follows:

$ hive --hiveconf hive.exec.max.created.files=<set the new limit>
answered May 21, 2019 by Charuni

Related Questions In Big Data Hadoop

0 votes
1 answer

How can we list files in HDFS directory as per timestamp?

No, there is no other option to ...READ MORE

answered May 8, 2018 in Big Data Hadoop by nitinrawat895
• 10,920 points
3,043 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 69,040 points
1,478 views
0 votes
1 answer

How to create files in hdfs?

You can create a file directly in ...READ MORE

answered Jan 25, 2019 in Big Data Hadoop by Omkar
• 69,040 points
262 views
0 votes
1 answer

How to create a Hive Database and store it in HDFS?

You will have to exclusively mention the ...READ MORE

answered May 20, 2019 in Big Data Hadoop by Jishnu
221 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,478 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
807 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
33,960 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,310 points
2,066 views
0 votes
1 answer

How do I create a subquery in hive?

Hey, Hive subquery is a select expression enclosed ...READ MORE

answered May 17, 2019 in Big Data Hadoop by Gitika
• 33,450 points
689 views
0 votes
1 answer

How can we configure remote metastore mode in Hive?

Hey,  Basically, hive-site.xml file has to be configured ...READ MORE

answered Jun 18, 2019 in Big Data Hadoop by Gitika
• 33,450 points
318 views