How to integrate Hadoop with Elasticsearch?

0 votes
I want to integrate hadoop with elastic search so, for this I have set up a Hadoop cluster with 3 DataNodes and 1 NameNode with elastic search installed on one of the DataNodes. But I’m not able to access HDFS using elastic search. Some one tells me that there is a separate plugin for hadoop and elastic search. Can someone tell me which plugin should I download and steps to integrate hadoop with elastic search.

Thanks in advance!
Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,010 points
118 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

The HDFS snapshot/restore plugin comes in three flavors:

  • Default / Hadoop 1.x The default version contains the plugin jar alongside Hadoop 1.x (stable) dependencies
  • Yarn / Hadoop 2.x The hadoop2 version contains the plugin jar plus the Hadoop 2.x (Yarn) dependencies.
  • Light The light version contains just the plugin jar, without any Hadoop dependencies.

What version to use?

It depends on whether you have Hadoop installed on your nodes or not. If you do, then it may be easiest to expose Hadoop to the Elasticsearch classpath (typically by creating symbolic links to the client jars within the Elasticsearch lib/ directory) and using the light version. This guarantees the existing libraries and configuration are being picked up by the plugin. If you do not have Hadoop installed, then select either the default version (for Hadoop stable/1.x) or, if you are using Hadoop 2, the hadoop2 version.

Below are the two repo links given which will help you to setup Hadoop with Elasticsearch.

https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs

https://github.com/elastic/elasticsearch-hadoop

answered Mar 21, 2018 by nitinrawat895
• 9,610 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,010 points
53 views
0 votes
1 answer

How to practice programming with Hadoop?

Well there are multiple ways to solve ...READ MORE

answered Mar 29, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
117 views
0 votes
1 answer

How to get started with Hadoop and do some development using Eclipse IDE?

Alright, there are couple of things that ...READ MORE

answered Apr 4, 2018 in Big Data Hadoop by Ashish
• 2,630 points
59 views
0 votes
1 answer

How to work with distributed cache in Hadoop?

The problem with your code is that ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
327 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
675 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,610 points
1,872 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
9,348 views
0 votes
1 answer
0 votes
1 answer

How to install Hadoop on Ubuntu?

You can refer to this blog by ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 9,610 points
272 views
0 votes
7 answers

How to run a jar file in hadoop?

I used this command to run my ...READ MORE

answered Dec 10, 2018 in Big Data Hadoop by Dasinto
3,689 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.