How to fix corrupt files on HDFS

0 votes

How does someone fix an HDFS that's corrupt? I looked on the Apache/Hadoop website and it said its fsck command, which doesn't fix it. Hopefully, someone who has run into this problem before can tell me how to fix this.

Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects. Normally NameNode automatically corrects most of the recoverable failures.

When I ran bin/Hadoop fsck / -delete, it listed the files that were corrupt or missing blocks. How do I make it not corrupt? This is on a practice machine so I COULD blow everything away but when we go live, I won't be able to "fix" it by blowing everything away so I'm trying to figure it out now.

Jul 18, 2019 in Big Data Hadoop by nitinrawat895
• 11,380 points
880 views

1 answer to this question.

0 votes

1 - Spark if following slave/master architecture. So on your cluster, you have to install a spark master and N spark slaves. You can run spark in a standalone mode. But using Yarn architecture will give you some benefits. There is a very good explanation of it here 

2- It is necessary if you want to use Yarn or HDFS for example, but as i said before you can run it in standalone mode.

answered Jul 18, 2019 by ravikiran
• 4,620 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to fix corrupt HDFS FIles?

You can use hdfs fsck / to ...READ MORE

answered Oct 25, 2018 in Big Data Hadoop by Frankie
• 9,830 points
2,782 views
0 votes
1 answer

How to count lines in a file on hdfs command?

Use the below commands: Total number of files: hadoop ...READ MORE

answered Aug 10, 2018 in Big Data Hadoop by Neha
• 6,300 points
26,869 views
0 votes
1 answer

How to discover missing or corrupt HDFS data?

HDFS supports fsck command to check for ...READ MORE

answered Sep 19, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,581 views
0 votes
1 answer

Hadoop HDFS: How to delete old files from HDFS?

You can use commands like this: hdfs dfs ...READ MORE

answered Nov 15, 2018 in Big Data Hadoop by Omkar
• 69,210 points
13,342 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,521 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,165 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
103,803 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,232 views
0 votes
1 answer

how to store images and videos on to HDFS?

HDFS is capable to accept data in ...READ MORE

answered Jul 16, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
2,525 views
0 votes
1 answer

Can anyone help me out to install the following packages R-MR, R-HDFS, and R-HBase on R-HAdoop?

I have understood your problem, I will ...READ MORE

answered May 31, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
484 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP