How many files can hive create in HDFS duringbMapReduce job?

0 votes
Is there a limit on how many files Hive can create on HDFS during a MapReduce job or can any number of files be created? I am writing a MapReduce program that involves creating files on HDFS. I wanted to know if there are any limits so that my program does not exceeds it.
May 21 in Big Data Hadoop by Sanam
17 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

There is a limit on how many files you can create on HDFS during MapReduce. The maximum number of files by default is set to 100000. But you can change this limit as follows:

$ hive --hiveconf hive.exec.max.created.files=<set the new limit>
answered May 21 by Charuni

Related Questions In Big Data Hadoop

0 votes
1 answer

How can we list files in HDFS directory as per timestamp?

No, there is no other option to ...READ MORE

answered May 8, 2018 in Big Data Hadoop by nitinrawat895
• 9,670 points
459 views
0 votes
1 answer

How to create a Hive table from sequence file stored in HDFS?

There are two SerDe for SequenceFile as ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Omkar
• 67,000 points
281 views
0 votes
1 answer

How to create files in hdfs?

You can create a file directly in ...READ MORE

answered Jan 25 in Big Data Hadoop by Omkar
• 67,000 points
63 views
0 votes
1 answer

How to create a Hive Database and store it in HDFS?

You will have to exclusively mention the ...READ MORE

answered May 20 in Big Data Hadoop by Jishnu
21 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,670 points
1,882 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,670 points
168 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
9,411 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
678 views
0 votes
1 answer

How do I create a subquery in hive?

Hey, Hive subquery is a select expression enclosed ...READ MORE

answered May 17 in Big Data Hadoop by Gitika
• 17,130 points
14 views
0 votes
1 answer

How can we configure remote metastore mode in Hive?

Hey,  Basically, hive-site.xml file has to be configured ...READ MORE

answered Jun 18 in Big Data Hadoop by Gitika
• 17,130 points
13 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.