Regarding unstructured data handling

0 votes
How can we import unstructured and semi structured data in Hadoop?
Jun 19 in Big Data Hadoop by Kunal
14 views

1 answer to this question.

0 votes
There are multiple ways to import unstructured data into Hadoop, depending on your use cases .

1. Using HDFS shell commands such as put or copyFromLocal to move flat files into HDFS. For details, please see File System Shell Guide.

2. Using WebHDFS REST API for application integration. WebHDFS REST API

3. Using Apache Flume. It is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS. Even though historically lots of use cases of Flume are involved with log data collection/aggregation, Flume can be used together with Kafka and turn itself into a real-time event processing pipeline.

4. Using Storm, a general-purpose, event-processing system. Within a topology composed of bolts and spouts, it can be used to ingest the event-based unstrcuture data into Hadoop

5. Spark's streaming component offers another alternative to ingest real-time unstructured data into the HDFS. Its processing model is quite different from Storm though. While Strom proceses incoming event one at a time, Spark streaming actually batches up events that arrive within a short time window before processing them. It is called mini-batch. Spark streaming of course runs on top of Spark Core computing engine, which is claimed to be 100x faster than MapReduce in memory and 10x faster on disk.
answered Jun 19 by Ranjitha

Related Questions In Big Data Hadoop

0 votes
1 answer

Doubt regarding sqoop updated data

Incremental append or load in sqoop will ...READ MORE

answered Feb 14 in Big Data Hadoop by Omkar
• 67,120 points
20 views
0 votes
1 answer

What is ClickStream Data Analysis

On a Web site, clickstream analysis (also ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
30 views
0 votes
1 answer
0 votes
1 answer

Is Kafka and Zookeeper are required in a Big Data Cluster?

Apache Kafka is one of the components ...READ MORE

answered Mar 22, 2018 in Big Data Hadoop by nitinrawat895
• 10,110 points
320 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,110 points
2,052 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,110 points
196 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,495 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
765 views
0 votes
1 answer

Regarding unstructured data handling in hadoop

There are multiple ways to import unstructured ...READ MORE

answered Jul 9 in Big Data Hadoop by Jishan
18 views
0 votes
1 answer

How will you choose various file formats for storing and processing data using Apache Hadoop ?

The decision to choose a particular file ...READ MORE

answered Sep 27, 2018 in Big Data Hadoop by zombie
• 3,690 points
67 views