What Distributed Cache is actually used for in Hadoop

0 votes
I am confused about the functionality of distributed cache? I have referred to multiple blogs but I still not able to properly understand what actually distibuted cache does?

Basically, I would like to know that by caching a file using distributed cache, is it stored or copied into every datanode? If not, does it mean copying the data into memory of each data node?
Apr 3, 2018 in Big Data Hadoop by Damon Salvatore
• 5,980 points
1,826 views

1 answer to this question.

0 votes
Basically distributed cache allows you to cache files that are need for your Map-Reduce jobs. Once you use distributed cache to cache a file for a job, it will be accessed by all the slave nodes (worker nodes) of the hadoop cluster. Basically, hadoop copies the cached files on the slave nodes' file system so that it can be accessed on that node. This copying of the file happens before any task w.r.t a job is executed on that node.

For further read and usage, you can refer to the official documentation: https://hadoop.apache.org/docs/r2.6.3/api/org/apache/hadoop/filecache/DistributedCache.html

Let me know in case you need anything else...
answered Apr 3, 2018 by Ashish
• 2,650 points

Related Questions In Big Data Hadoop

0 votes
1 answer

What is distributed Cache in MapReduce Framework ?

Distributed Cache is an important feature provided ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Frankie
• 9,830 points
1,325 views
0 votes
1 answer

I have to ingest in hadoop cluster large number of files for testing , what is the best way to do it?

Hi@sonali, It depends on what kind of testing ...READ MORE

answered Jul 8, 2020 in Big Data Hadoop by MD
• 95,440 points
935 views
0 votes
2 answers

What is Hadoop used for?

Apache Hadoop is an open-source framework that ...READ MORE

answered Dec 10, 2021 in Big Data Hadoop by CoderGirl
• 500 points
390 views
0 votes
1 answer
0 votes
1 answer

What is the function of getLocalCacheArchives method?

We use distributed cache to share those ...READ MORE

answered Apr 29, 2018 in Big Data Hadoop by Shubham
• 13,490 points
523 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,521 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,165 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
103,810 views
0 votes
1 answer

What is the use of sequence file in Hadoop?

Sequence files are binary files containing serialized ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by Ashish
• 2,650 points
9,176 views
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP