Regarding unstructured data handling in hadoop

0 votes
How can we import unstructured and semi-structured data in Hadoop? It is easy to import to structured data because I can directly import it from MySQL using Sqoop. But what to do in case of unstructured data?
Jul 9, 2019 in Big Data Hadoop by Yogi
4,451 views

1 answer to this question.

0 votes

There are multiple ways to import unstructured data into Hadoop, depending on your use cases.

1. Using HDFS shell commands such as put or copyFromLocal to move flat files into HDFS. For details, please see File System Shell Guide.

2. Using WebHDFS REST API for application integration. WebHDFS REST API

3. Using Apache Flume. It is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS. Even though historically lots of use cases of Flume are involved with log data collection/aggregation, Flume can be used together with Kafka and turn itself into a real-time event processing pipeline.

4. Using Storm, a general-purpose, event-processing system. Within a topology composed of bolts and spouts, it can be used to ingest the event-based unstructured data into Hadoop

5. Spark's streaming component offers another alternative to ingesting real-time unstructured data into the HDFS. Its processing model is quite different from Storm though. While Strom process incoming event one at a time, Spark streaming actually batches up events that arrive within a short time window before processing them. It is called a mini-batch. Spark streaming, of course, runs on top of Spark Core computing engine, which is claimed to be 100x faster than MapReduce in memory and 10x faster on disk.

To know more, It's recommended to join Big Data Certification today.

answered Jul 9, 2019 by Jishan

Related Questions In Big Data Hadoop

+1 vote
1 answer

Is Hadoop only Framework in Big Data Ecosystem ?

Actually there are many other frameworks, one of ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Ashish
• 2,650 points
753 views
0 votes
1 answer

Why Java Code in Hadoop uses own Data Types instead of basic Data types?

Hadoop provides us Writable interface based data ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
868 views
0 votes
1 answer

How to implement data locality in Hadoop MapReduce?

You can use this getFileBlockLocations method of ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
692 views
0 votes
1 answer

What is the Data format and database choices in Hadoop and Spark?

Use Parquet. I'm not sure about CSV ...READ MORE

answered Sep 4, 2018 in Big Data Hadoop by Frankie
• 9,830 points
715 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,555 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,184 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
104,185 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,260 views
0 votes
1 answer

Regarding unstructured data handling

There are multiple ways to import unstructured ...READ MORE

answered Jun 19, 2019 in Big Data Hadoop by Ranjitha
429 views
0 votes
1 answer

Hadoop Hive: How to insert data in Hive table?

First, copy data into HDFS. Then create ...READ MORE

answered Nov 12, 2018 in Big Data Hadoop by Omkar
• 69,210 points
9,448 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP