How to integrate Hadoop with Elasticsearch

0 votes
I want to integrate hadoop with elastic search so, for this I have set up a Hadoop cluster with 3 DataNodes and 1 NameNode with elastic search installed on one of the DataNodes. But I’m not able to access HDFS using elastic search. Some one tells me that there is a separate plugin for hadoop and elastic search. Can someone tell me which plugin should I download and steps to integrate hadoop with elastic search.

Thanks in advance!
Mar 22, 2018 in Big Data Hadoop by coldcode
• 2,080 points
1,208 views

1 answer to this question.

0 votes

The HDFS snapshot/restore plugin comes in three flavors:

  • Default / Hadoop 1.x The default version contains the plugin jar alongside Hadoop 1.x (stable) dependencies
  • Yarn / Hadoop 2.x The hadoop2 version contains the plugin jar plus the Hadoop 2.x (Yarn) dependencies.
  • Light The light version contains just the plugin jar, without any Hadoop dependencies.

What version to use?

It depends on whether you have Hadoop installed on your nodes or not. If you do, then it may be easiest to expose Hadoop to the Elasticsearch classpath (typically by creating symbolic links to the client jars within the Elasticsearch lib/ directory) and using the light version. This guarantees the existing libraries and configuration are being picked up by the plugin. If you do not have Hadoop installed, then select either the default version (for Hadoop stable/1.x) or, if you are using Hadoop 2, the hadoop2 version.

Below are the two repo links given which will help you to setup Hadoop with Elasticsearch.

https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs

https://github.com/elastic/elasticsearch-hadoop

answered Mar 22, 2018 by nitinrawat895
• 11,380 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,080 points
899 views
0 votes
1 answer

How to practice programming with Hadoop?

Well there are multiple ways to solve ...READ MORE

answered Mar 30, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,650 views
0 votes
1 answer

How to get started with Hadoop and do some development using Eclipse IDE?

Alright, there are couple of things that ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by Ashish
• 2,650 points
1,738 views
0 votes
1 answer

How to work with distributed cache in Hadoop?

The problem with your code is that ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
1,164 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,260 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,555 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
104,199 views
0 votes
1 answer
0 votes
1 answer

How to install Hadoop on Ubuntu?

You can refer to this blog by ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,188 views
0 votes
7 answers

How to run a jar file in hadoop?

I used this command to run my ...READ MORE

answered Dec 10, 2018 in Big Data Hadoop by Dasinto
25,530 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP