How to integrate Hadoop with Elasticsearch?

0 votes
I want to integrate hadoop with elastic search so, for this I have set up a Hadoop cluster with 3 DataNodes and 1 NameNode with elastic search installed on one of the DataNodes. But I’m not able to access HDFS using elastic search. Some one tells me that there is a separate plugin for hadoop and elastic search. Can someone tell me which plugin should I download and steps to integrate hadoop with elastic search.

Thanks in advance!
Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
238 views

1 answer to this question.

0 votes

The HDFS snapshot/restore plugin comes in three flavors:

  • Default / Hadoop 1.x The default version contains the plugin jar alongside Hadoop 1.x (stable) dependencies
  • Yarn / Hadoop 2.x The hadoop2 version contains the plugin jar plus the Hadoop 2.x (Yarn) dependencies.
  • Light The light version contains just the plugin jar, without any Hadoop dependencies.

What version to use?

It depends on whether you have Hadoop installed on your nodes or not. If you do, then it may be easiest to expose Hadoop to the Elasticsearch classpath (typically by creating symbolic links to the client jars within the Elasticsearch lib/ directory) and using the light version. This guarantees the existing libraries and configuration are being picked up by the plugin. If you do not have Hadoop installed, then select either the default version (for Hadoop stable/1.x) or, if you are using Hadoop 2, the hadoop2 version.

Below are the two repo links given which will help you to setup Hadoop with Elasticsearch.

https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs

https://github.com/elastic/elasticsearch-hadoop

answered Mar 21, 2018 by nitinrawat895
• 10,730 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
106 views
0 votes
1 answer

How to practice programming with Hadoop?

Well there are multiple ways to solve ...READ MORE

answered Mar 29, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
221 views
0 votes
1 answer

How to get started with Hadoop and do some development using Eclipse IDE?

Alright, there are couple of things that ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by Ashish
• 2,630 points
131 views
0 votes
1 answer

How to work with distributed cache in Hadoop?

The problem with your code is that ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
436 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
1,227 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,730 points
3,374 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
16,806 views
0 votes
1 answer
0 votes
1 answer

How to install Hadoop on Ubuntu?

You can refer to this blog by ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 10,730 points
369 views
0 votes
7 answers

How to run a jar file in hadoop?

I used this command to run my ...READ MORE

answered Dec 10, 2018 in Big Data Hadoop by Dasinto
6,129 views